LGPL.txt0100644000076400001440000006347607727073100011226 0ustar gabiusers GNU LESSER GENERAL PUBLIC LICENSE Version 2.1, February 1999 Copyright (C) 1991, 1999 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. [This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library Public License, version 2, hence the version number 2.1.] Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. We call this license the "Lesser" General Public License because it does Less to protect the user's freedom than the ordinary General Public License. It also provides other free software developers Less of an advantage over competing non-free programs. These disadvantages are the reason we use the ordinary General Public License for many libraries. However, the Lesser license provides advantages in certain special circumstances. For example, on rare occasions, there may be a special need to encourage the widest possible use of a certain library, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use the library. A more frequent case is that a free library does the same job as widely used non-free libraries. In this case, there is little to gain by limiting the free library to free software only, so we use the Lesser General Public License. In other cases, permission to use a particular library in non-free programs enables a greater number of people to use a large body of free software. For example, permission to use the GNU C Library in non-free programs enables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linux operating system. Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that the user of a program that is linked with the Library has the freedom and the wherewithal to run that program using a modified version of the Library. The precise terms and conditions for copying, distribution and modification follow. Pay close attention to the difference between a "work based on the library" and a "work that uses the library". The former contains code derived from the library, whereas the latter must be combined with the library in order to run. GNU LESSER GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any software library or other program which contains a notice placed by the copyright holder or other authorized party saying it may be distributed under the terms of this Lesser General Public License (also called "this License"). Each licensee is addressed as "you". A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application programs (which use some of those functions and data) to form executables. The "Library", below, refers to any such software library or work which has been distributed under these terms. A "work based on the Library" means either the Library or any derivative work under copyright law: that is to say, a work containing the Library or a portion of it, either verbatim or with modifications and/or translated straightforwardly into another language. (Hereinafter, translation is included without limitation in the term "modification".) "Source code" for a work means the preferred form of the work for making modifications to it. For a library, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the library. Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running a program using the Library is not restricted, and output from such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. 1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) The modified work must itself be a software library. b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change. c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License. d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful. (For example, a function in a library to compute square roots has a purpose that is entirely well-defined independent of the application. Therefore, Subsection 2d requires that any application-supplied function or table used by this function must be optional: if the application does not supply it, the square root function must still compute square roots.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Library, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Library, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Library. In addition, mere aggregation of another work not based on the Library with the Library (or with a work based on the Library) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices. Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General Public License applies to all subsequent copies and derivative works made from that copy. This option is useful when you wish to copy part of the code of the Library into a program that is not a library. 4. You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange. If distribution of object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place satisfies the requirement to distribute the source code, even though third parties are not compelled to copy the source along with the object code. 5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. However, linking a "work that uses the Library" with the Library creates an executable that is a derivative of the Library (because it contains portions of the Library), rather than a "work that uses the library". The executable is therefore covered by this License. Section 6 states terms for distribution of such executables. When a "work that uses the Library" uses material from a header file that is part of the Library, the object code for the work may be a derivative work of the Library even though the source code is not. Whether this is true is especially significant if the work can be linked without the Library, or if the work is itself a library. The threshold for this to be true is not precisely defined by law. If such an object file uses only numerical parameters, data structure layouts and accessors, and small macros and small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardless of whether it is legally a derivative work. (Executables containing this object code plus portions of the Library will still fall under Section 6.) Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under the terms of Section 6. Any executables containing that work also fall under Section 6, whether or not they are linked directly with the Library itself. 6. As an exception to the Sections above, you may also combine or link a "work that uses the Library" with the Library to produce a work containing portions of the Library, and distribute that work under terms of your choice, provided that the terms permit modification of the work for the customer's own use and reverse engineering for debugging such modifications. You must give prominent notice with each copy of the work that the Library is used in it and that the Library and its use are covered by this License. You must supply a copy of this License. If the work during execution displays copyright notices, you must include the copyright notice for the Library among them, as well as a reference directing the user to the copy of this License. Also, you must do one of these things: a) Accompany the work with the complete corresponding machine-readable source code for the Library including whatever changes were used in the work (which must be distributed under Sections 1 and 2 above); and, if the work is an executable linked with the Library, with the complete machine-readable "work that uses the Library", as object code and/or source code, so that the user can modify the Library and then relink to produce a modified executable containing the modified Library. (It is understood that the user who changes the contents of definitions files in the Library will not necessarily be able to recompile the application to use the modified definitions.) b) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (1) uses at run time a copy of the library already present on the user's computer system, rather than copying library functions into the executable, and (2) will operate properly with a modified version of the library, if the user installs one, as long as the modified version is interface-compatible with the version that the work was made with. c) Accompany the work with a written offer, valid for at least three years, to give the same user the materials specified in Subsection 6a, above, for a charge no more than the cost of performing this distribution. d) If distribution of the work is made by offering access to copy from a designated place, offer equivalent access to copy the above specified materials from the same place. e) Verify that the user has already received a copy of these materials or that you have already sent this user a copy. For an executable, the required form of the "work that uses the Library" must include any data and utility programs needed for reproducing the executable from it. However, as a special exception, the materials to be distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute. 7. You may place library facilities that are a work based on the Library side-by-side in a single library together with other library facilities not covered by this License, and distribute such a combined library, provided that the separate distribution of the work based on the Library and of the other library facilities is otherwise permitted, and provided that you do these two things: a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities. This must be distributed under the terms of the Sections above. b) Give prominent notice with the combined library of the fact that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. 8. You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 9. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Library or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Library (or any work based on the Library), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Library or works based on it. 10. Each time you redistribute the Library (or any work based on the Library), the recipient automatically receives a license from the original licensor to copy, distribute, link with or modify the Library subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties with this License. 11. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Library at all. For example, if a patent license would not permit royalty-free redistribution of the Library by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Library. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply, and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 12. If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Library under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 13. The Free Software Foundation may publish revised and/or new versions of the Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Library specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Library does not specify a license version number, you may choose any version ever published by the Free Software Foundation. 14. If you wish to incorporate parts of the Library into other free programs whose distribution conditions are incompatible with these, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Libraries If you develop a new library, and you want it to be of the greatest possible use to the public, we recommend making it free software that everyone can redistribute and change. You can do so by permitting redistribution under these terms (or, alternatively, under the terms of the ordinary General Public License). To apply these terms, attach the following notices to the library. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Also add information on how to contact you by electronic and paper mail. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the library, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the library `Frob' (a library for tweaking knobs) written by James Random Hacker. , 1 April 1990 Ty Coon, President of Vice That's all there is to it! Makefile0100644000076400001440000000202710230226350011336 0ustar gabiusers# #/* # The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. # Version 0.9.x (patchlevel see file Version) # Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 2.1 of the License, or (at your option) any later version. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # see README for more information #EOH */ # # $Id: Makefile,v 1.2 2005/01/24 18:23:21 kripke Exp $ # the Makefile # all: make -C mlt Mingw.mk0100644000076400001440000000013710233727727011331 0ustar gabiusersPATH:=/opt/cross-tools/mingw-3/i386-pc-mingw32/bin:/opt/cross-tools/mingw-3/bin:$(PATH) CC=gcc README0100644000076400001440000000032710233236705010567 0ustar gabiusersmake -C mlt server then check out php/demo.php or telnet localhost 2042 and type (be careful to use a tab after the Q with queries!): test.Q test.Q test.Q water test.Q plant water? test.Q plant + dev$ test.Q Version0100644000076400001440000000473410240340227011256 0ustar gabiusers1.0.2 LGPL.txt 1.1 Mon Sep 8 12:29:52 2003 Makefile 1.2 Sat Apr 16 15:19:04 2005 Mingw.mk 1.1 Mon Sep 8 12:34:50 2003 README 1.4 Mon Apr 25 18:58:13 2005 Version 1.13 Wed May 11 08:21:08 2005 doc/.index 1.9 Sat May 7 11:48:02 2005 doc/CDS.txt 1.3 Thu Jul 22 12:45:31 2004 doc/CharSet.txt 1.12 Fri Nov 12 11:18:23 2004 doc/CmdLine.txt 1.6 Fri Nov 12 11:30:17 2004 doc/Converting.txt 1.1 Thu Sep 23 11:47:20 2004 doc/Diff09.txt 1.2 Thu Apr 8 15:17:51 2004 doc/DownLoad.txt 1.16 Wed May 11 08:24:29 2005 doc/FileFormats.txt 1.7 Fri Nov 12 11:18:23 2004 doc/IIF.txt 1.5 Thu Sep 23 11:44:04 2004 doc/MOM.txt 1.3 Mon May 3 13:04:36 2004 doc/MetaData.txt 1.2 Mon Jul 26 13:14:25 2004 doc/MultiProcess.txt 1.5 Thu Apr 8 15:17:51 2004 doc/OverView.txt 1.4 Tue Jun 15 13:17:37 2004 doc/Protocol.txt 1.13 Sat Apr 16 15:19:19 2005 doc/Query.txt 1.19 Fri Jul 16 12:42:39 2004 doc/RecStruct.txt 1.6 Mon Jul 26 12:23:34 2004 doc/Selene.txt 1.1 Sat May 7 11:47:17 2005 doc/Status.txt 1.4 Sat Apr 16 15:19:19 2005 doc/TagUse.txt 1.2 Mon Jul 26 13:20:18 2004 doc/Tcl.txt 1.1 Thu Sep 23 11:45:43 2004 mlt/Makefile 1.2 Wed Apr 27 15:43:17 2005 mlt/cds.c 1.1 Mon Jan 24 17:58:35 2005 mlt/cdx.c 1.1 Mon Jan 24 17:58:49 2005 mlt/core.h 1.2 Wed Feb 23 17:09:51 2005 mlt/db.c 1.1 Mon Jan 24 17:59:47 2005 mlt/dbo.c 1.2 Tue Apr 26 22:40:54 2005 mlt/hash.c 1.1 Mon Jan 24 18:00:20 2005 mlt/iif.c 1.1 Mon Jan 24 18:00:40 2005 mlt/malete.c 1.2 Mon Apr 25 18:58:31 2005 mlt/pw.c 1.1 Mon Jan 24 18:01:20 2005 mlt/pw.h 1.2 Wed Feb 23 17:09:52 2005 mlt/qdx.c 1.2 Mon Apr 25 18:58:31 2005 mlt/rdx.c 1.1 Mon Jan 24 18:03:28 2005 mlt/run 1.1 Mon Jan 24 18:04:00 2005 mlt/srv.c 1.1 Mon Jan 24 18:04:11 2005 mlt/sys.c 1.2 Mon Apr 25 18:58:31 2005 mlt/tool.h 1.3 Wed May 11 08:22:02 2005 mlt/uti.c 1.1 Mon Jan 24 18:04:54 2005 php/Isis.php 1.1 Wed Mar 17 13:26:42 2004 php/Isis/Db.php 1.2 Tue Nov 2 13:44:24 2004 php/Isis/Http.php 1.2 Wed Nov 3 13:56:57 2004 php/Isis/Rec.php 1.2 Wed Nov 3 13:57:15 2004 php/Isis/Server.php 1.4 Fri Nov 12 09:34:55 2004 php/demo.php 1.3 Tue Nov 2 13:44:42 2004 php/killidx.php 1.1 Tue Apr 26 13:56:44 2005 php/killqry.php 1.1 Tue Apr 26 22:27:55 2005 php/unicode.php 1.3 Fri Nov 12 09:35:17 2004 test/cds.m0d 1.3 Thu Nov 11 18:19:55 2004 test/cds.mqt 1.1 Fri Jan 30 15:18:11 2004 test/cds.mrd 1.2 Fri Apr 2 11:54:35 2004 test/unicode.m0d 1.2 Wed Nov 3 17:41:46 2004 test/unicode.mqt 1.1 Wed Nov 3 13:58:43 2004 test/unicode.mrd 1.1 Wed Nov 3 13:58:43 2004 doc/.index0100644000076400001440000000063010237125362011561 0ustar gabiusers! > http://openisis.org/Doc/ Isi * Selene > Selene Announcement * Malete > DownLoad > Status > OverView > CmdLine Usage > RecStruct Structures > Protocol > Query > MultiProcess * formats > FileFormats > CharSet character sets > CDS CDS/ISIS > IIF IIF/ISO2709 * OpenIsis > http://openisis.org/Doc/ still here doc/CDS.txt0100644000076400001440000000237510077733353011642 0ustar gabiusersWorking with CDS/ISIS databases. A couple of databases have been converted successfully from/to WinIsis. If CDSUNIX is defined during make, it will use the "UNIX" format as used by wwwisis on Unix machines. Usage: $ bin/malete cdsimp cds -ftest/ $ will look for test/cds.mst and the like and write a Malete database at path test/cds. The Malete database files should not exist, else records there with like MFNs will be updated. $ bin/malete cdsexp cds -ftest/ $ does the reverse. You can try with the provided test/cds, after creating its index using $ bin/malete qload cds -ftest/ http://ietf.org/rfc/rfc2278.txt CCS + CES ) which is compatible with > http://ietf.org/rfc/rfc0020.txt ASCII so that - every character defined by ASCII is encoded by the same byte value - every byte with a value in the range 0-127 inclusive encodes the character as specified by ASCII This - includes > http://aspell.net/charsets/iso8859.html ISO-8859-* , Unicode in the > http://www.ietf.org/rfc/rfc2279.txt UTF-8 encoding, various far east > http://web.archive.org/web/czyborra.com/utf/#EUC EUC and similar encodings usings pairs of bytes greater than 127 and works well with most (but not all) IBM/M$ > http://aspell.net/charsets/codepages.html codepages unless you really abused the control characters 0-31 for graphics - excludes > http://web.archive.org/web/czyborra.com/utf/#UTF-16 UTF-16 and other formats using bytes 0-127 as part of multibyte sequences and, of course, the anti-ASCII > http://aspell.net/charsets/iso646.html#EBCDIC EBCDIC - should work with some restrictions on searching for encodings which at least preserve linefeed (10), horizontal TAB (9) and the digits (48-57) like the Unicode standard compression scheme > http://www.unicode.org/unicode/reports/tr6/ SCSU , > http://aspell.net/charsets/vietnamese.html VISCII and even old > http://aspell.net/charsets/iso646.html#i18n ISO-646-* or Cyrillic > http://aspell.net/charsets/cyrillic.html KOI In order to store and retrieve (by record id) data, Malete does not need to know anything about the character set. However, the > MetaData content-type may contain a charset name using a preferred MIME name from the > http://www.iana.org/assignments/character-sets iana registered charsets. The basic server does not support character set conversion, since many client environments like Java or Tcl are well prepared to handle this. A Tcl based server may support charset conversion. * indexing For indexing, we use quite a lot of information about characters similar to, but extending the traditional > http://www.cindoc.csic.es/isis/21-5.htm ISISUC.TAB and > http://www.cindoc.csic.es/isis/21-6.htm ISISAC.TAB - a sort order (collation sequence), possibly using > http://oss.software.ibm.com/icu/userguide/Collate_Concepts.html multilevel comparison according to the > http://unicode.org/unicode/reports/tr10/ UCA - a mapping of certain code sequences to others, for example to use uppercased versions in the index - some notion of which characters are "alphabetic" (parts of words) for indexing in word split mode This data is configured by a sequence of strings, which may be obtained from the collation (4) fields of a database's > MetaData metadata record (in _database_.m0d) or as the lines of a textfile (_collation_.mcd, not implemented). These strings start with a one or two character mnemonic which is followed by a tabulator separated sequences of bytes, representing single characters or sequences of characters in the database's encoding. Unlike CDS/ISIS, Malete always deals with multibyte entitities and does not use explicit codes as decimal numbers. Consequently, the collation configuration can be converted between charsets just like the database itself (e.g. using recode or iconv). Malete does not care whether a multibyte sequence holds the two ASCII characters 'C' and 'H' in order to assign 'CH' a separate rank between 'C' and 'D' in spanish collation (a > http://oss.software.ibm.com/icu/userguide/Collate_Concepts.html#Contractions Contraction ) or an > http://www.loc.gov/marc/specifications/speccharmarc8.html ANSEL ( > http://lcweb2.loc.gov/cocoon/codetables/45.html codes , > http://www.niso.org/standards/resources/Z39-47.pdf Z39.47 ) style composition or UTF-8 using two or more bytes to encode a character with a diacritical mark (in precomposed or decomposed form). Configuration entries supported in the initial version are: - collation C _name_ [_options_] assigns a name to this collation or refers to an external collation. Only the first 31 bytes in _name_ are considered. Should be a C identifier (plain ASCII) for best interoperability. Proposed (but probably not implemented) options are 'c' for compression and 'f' for french (reverse) secondaries (see below). - word W _entities_ specifies that the listed entities are considered parts of words and assigns sort ranks in ascending order to them - nonword N _entities_ like W, but the entities separate "words" in word split mode. Multiple W and N entries can be used to assign successive sort ranks. - alias A _entities_ the entities are assigned as aliases to the corresponding entities of the last seen W or N, e.g. a sequence of lowercase characters to their uppercase equivalents - map M _entities_ the second and following entities are mapped to first one, which will be iteratively checked for other rules (but not maps). This can be used to map entities to empty (completey discarding them) or to multiple entities as an > http://oss.software.ibm.com/icu/userguide/Collate_Concepts.html#Expansions Expansion Example:) $ 4 C spanish_de_PHONEBOOK 4 N . , ; - _ 4 N 0 1 2 3 4 5 6 7 8 9 4 W a b c ch d e f g h ... 4 A A B C CH D E F G H ... 4 A 4 A 4 M ae 4 M oe $ Here, 'coche' sorts exactly like 'COCHE' after 'cocina', since the ch sorts after the c (and the i is not even considered). 'Knig' in german phonebooks sorts exactly like 'Koenig', and in a terms listing, both will be displayed as 'koenig'. (Unlike > http://oss.software.ibm.com/icu/charts/collation/de__PHONEBOOK.html "de_PHONEBOOK" , the > http://oss.software.ibm.com/icu/charts/collation/de.html "de" collation has the o-umlaut as secondary to o). Note that as a possibly confusing, although correct sideeffect, a prefix search for 'coc' will NOT match 'coche', since it does not beginn with the codes for 'c'-'o'-'c', but with those for 'c'-'o'-'ch'. * implementation The collation is basically implemented by means of a recoding. Every W and N entity (byte sequence) corresponds to one code number, increasing from 2 in the order of their definition. Every unrecognized byte value, and especially the TAB (9), which can not be redefined, maps to code number 1 and runs of those are squeezed (i.e. only one 1 for a sequence of unrecognized bytes). Aliases use the same code number as their corresponding W or N entity. The recoded key is then a sequence of code numbers corresponding to the recognized entities. Depending on the highest code number, one or two bytes (big endian) are used for every number. This transformation is applied to every index entry before storing it, as well as to every term before lookup. From the table of entities, the original term (in W and N entities, not aliases or maps) can be reconstructed for display of indexed terms. Note that the term decoded from a collation key does not necessarily map to the same key. Where their byte sequences overlap with others, they may become parts of other contractions. The implementation limits both the length (in bytes) of sequences and the number of codes of a map target to 15. * compression With the compression option implemented and enabled, the number of bits used per code is the minimal number of bits needed to represent the highest code number, and the bitstring is padded with 0 bits to the full byte. In the spanish environment one would need 29 alphabetic codes (including CH, LL and ), 10 digits and some punctuation, so six bits (codes 2-63) are sufficient and we can reduce key size by up to 25%. This probably is especially interesting for databases integrating a lot of phoenician and/or brahmi scripts, using more than 256 but less than 512 codes. Here one would need only 9 instead of 16 bits, saving more than 40%. In a CJK environment, you will need at least 15 bits anyway. Do not confuse this compression of single keys with the option of > FileFormats compressing the index based on common prefixes between adjacent keys. * multilevel comparison Some future version will also support S and T entries to support secondary (optionally french) and tertiary levels, possibly one day even quaternary and identical levels Q and I, should there be demand. Example (c.f. the > http://oss.software.ibm.com/icu/charts/collation/es.html es chart , lacking the ch contraction): $ 4 C spanish 4 W a b c ch d e f 4 T A B C CH D E F 4 S 4 T 4 S 4 T $ Here, 'coche' sorts before 'Coche', since on the third level the 'c' sorts before 'C'. (Unlike plain ASCII sorting, most collations sort lowercase before uppercase). Still 'coche' sorts after 'Cocina', since the primary difference between 'ch' and 'c' takes precedence over the tertiary difference, although the latter occurs earlier in the word. Just for the fun of it, the a-umlaut is not expanded here, but listed as another secondary variant of 'a' with it's own tertiary. For multilevel comparison, a 0 code plus additional bits are appended to the recoded key. First, for every character some bits are appended to code it's secondary variant, depending on how many variants are defined for the character, then likewise for tertiary variants. In Latin scripts, typically every alphabetical character has one tertiary variant (it's lowercase equivalent, using one bit) and some or all vowels can have one or more diacritical marks. By appending additional bits not only do terms sort properly, but moreover we have the option for an exact match sensitive to all levels or a match insensitive to third or second and third level very similar to a prefix match (since the first level IS a prefix). An actual prefix match should usually be done using only the first level bits, checking for second and third level prefix is a little bit more complicated. For french secondary sorting, the second level bits are appended in reverse order. Must not be used together with left-to-right secondary sorting. Using the additional bits, a terms listing can reconstruct the input with regard to all variants, i.e. with proper case and diacritics. However, aliases and mappings can not be reversed: where the a umlaut should sort *exactly* like an a followed by an e, it uses exactly the same bytes and we can not tell from the index that once there was an a umlaut in the input. * east asian word indexing Segmentation of > http://www.multilingual.com/FMPro?-db=archives&-format=ad%2fselected%5fresults.html&-find= Chinese text is, in general, not a trivial task. Of course, the use of spaces to explicitly separate words is an option. The usual word split will just work as for any other scripts. Since Malete support full application controlled indexing, it is also possible to use any existing segmentation algorithm on the application level. Where this is unwanted or not possible, a somewhat brute force, yet feasible approach is to put every single character or every digraph or trigraph in the index. (I.e. every character together with the two or three following characters). Please contact us, should there be demand for such a character or m-graph indexing method as alternative to "word" split. * using collation When a database is opened, Malete looks for the file _database_.m0d (where _database_ is the name yo your database, e.g. cds.m0d). If this exists, it is scanned for 4 fields. If there is a "4 C" naming the collation _name_, and there either is a compiled collation file _name_.mcx or another database is already using a collation of that name, the existing collation info is used. Else a new collation (with the given name or anonymous) is created from the description in the metadata and, if it is named, saved as file _name_.mcx (in the current directory). The distribution contains two sample collation definitions, a Latin-1 based as test/cds.m0d and a UTF-8 (Unicode) based as test/unicode.m0d. Use a UTF-8 capable editor like "vim '+set encoding=utf-8'" in a "xterm -u8". Creating a collation definition from existing ISISAC.TAB and ISISUC.TAB is a straightforward exercise left to the reader. - *WARNING*: Adding or changing the collation in _database_.m0d will render your encoded index data garbage! Make sure to save the plaintext index data using "malete qdump _database_ >_database_.mqt" before and reencode the index with "malete qload _database_ <_database_.mqt" after editing the m0d file. - *WARNING*: While named (and thus shared) collations are much more efficient, multiple databases using a different specification for the same collation name will not properly coexist. When changing a named collation, be sure to remove the .mcx file and reload the indexes for all affected databases. When in doubt, remove or change the collation name. * links > http://openisis.org/Doc/encoding list of encodings supported by Java > http://openisis.org/Doc/Unicode notes on charsets and unicode with ISIS > http://openisis.org/Doc/CsTables tables of some western latin sets > http://openisis.org/Doc/UniStats statistics on unicode character properties > http://openisis.org/Doc/Collating approaches to collation > http://www.iana.org/assignments/character-sets iana registered charsets > http://web.archive.org/web/czyborra.com/unicode/characters.html#scripts unicode characters and scripts ICU Collation > http://oss.software.ibm.com/icu/userguide/Collate_Intro.html Introduction and > http://oss.software.ibm.com/icu/userguide/Collate_Concepts.html Concepts Collation charts (without contractions) by > http://oss.software.ibm.com/icu/charts/collation/ locale or > http://www.unicode.org/charts/collation/ script (according to Unicode > http://www.unicode.org/unicode/reports/tr10/tr10-11.html#Default_Unicode_Collation_Element_Table default collation ) --- $Id: CharSet.txt,v 1.12 2004/11/12 11:18:23 kripke Exp $ doc/CmdLine.txt0100644000076400001440000000605510145117311012525 0ustar gabiusers;man malete 1 $Date: 2004/11/12 11:30:17 $ Malete tool command line options. * synopsis The general format is $ malete command [general and command options] [db0 opts [db1 opts ...]] $ Every parameter not starting with a '-' becomes the main value of a new field in the environment record. Every option (a parameter starting with a '-') is turned into a subfield of the field introduced by the last non-option. The first character is the subfield identifier, the remainder is the subfields value. Command and the options following it control the overall process behaviour, other fields specify options for objects (i.e. databases, in the future also other objects like remote servers). Example: $ malete rdump -vi cds -ftest/ $ dumps records of a database named cds in files test/cds.m[rq][dx] to stdout and some informational messages to stderr. * general options General environment options use lowercase subfield identifiers (see sys.c): - vn sets the logging level. n is a digit from 0 (off) to 9 (all). You may also use the first letter of off, fatal, syserr, err, warn, info, verbose, debug, trace or all. - wn sets the writing/sharing mode. default is exclusive write mode. n='0' selects shared read-only, n='s' selects shared writing (not supported on windows). * database options When a database is opened, it scans the environment record for a field with the database name as main value for options (see db.c): - b selects blocking. The process will wait for the shared or exclusive file lock. Without this option, it will complain immediatly. - fpath specifies db files. If path ends with /, it is used as directory, else as file basename (w/o ext). - qpn specifies the index type. A single q turns use of the index off (i.e. no index files are created). Else p is a letter in klpt, setting the keysize, let, pointer type or tree type, resp., to number n. Use only if you understood the > FileFormats . For example _-qk30_ will set the keysize to 30. - rn sets the record xref type to number n. A single r turns record file use off. - wm sets the write mode. m='0' selects read-only, m='a' selects asynchronous write (recommended only when importing databases). No effect in a read-only environment. * simple commands A couple of simple commands have no specific options. Most commands require one database to be specified as target. - dbinfo print database parameters - qdump dump index to stdout as lines of tab-separated key,rid,tag,occ,pos. - qload load index from stdin in the same format - rdump dump records - rload load records * command options The various commands use uppercase letters for command specific options. - server runs a server Without any options, reads from stdin and answers on stdout. The -S option specifies a TCP port or UNIX (absolute filename) socket. The TCP port may also be prefixed by "localhost:" to bind locally. Without any value, -S uses TCP port 2042 (registered by kchapman@isis.com). - cdsimp or cdsexp import or export from/to > CDS /ISIS databases. - iifimp or iifexp import or export from/to > IIF (ISO2709) files. doc/Converting.txt0100644000076400001440000000210710124533710013324 0ustar gabiusersconverting between character sets * overview Since Malete can represent both record and index data as plain text files, standard tools like > http://www.gnu.org/software/recode/recode.html recode or > http://www.gnu.org/software/libiconv/ iconv can be used for character set conversion. This can also be used to conver WinIsis or wwwisist files via > CDS cds import/export. * example To convert the standard WinIsis cds database to malete as is, without any character set conversion, use $ malete cdsimp -No850 cds $ Create a textual representation of the index by $ malete qdump cds 2>cds.mqt $ Convert the textfiles to UTF-8 use $ iconv -f CP850 -t UTF-8 cds.mrd-u iconv -f CP850 -t UTF-8 cds.mqt-u $ Replace the Malete database with the UTF-8 files $ rm cds.mq? cds.mr? mv cds.mrd-u cds.mrd malete qload cds /tar/malete.1.0.2.tgz sources , > /tar/malete.1.0.2.gz linux binary (statically linked) , > /tar/malete102.zip windows binary * 1.0 Contains collation (ordering) support (including Unicode) as described in > CharSet character sets. Also a couple of bugs have been fixed. Both the basic and unicode PHP demos have been successfully tested against malete server on both Win95 and XP. NOTE: Be sure to read and understand "using collation" in > CharSet character sets before playing around with it. NOTE: The preliminary Latin-1 character mapping is gone with this release. If there is no collation info, data is indexed without any changes! This is the last feature release (equal to the 0.9.9e release candidate). > /tar/malete.1.0.tgz sources > /tar/malete.0.9.9e.gz linux binary (statically linked) > /tar/malete099e.zip windows binary * 0.9.9d Fixes CDS import of "locked" records. > /tar/malete.0.9.9d.tgz sources > /tar/malete.0.9.9d.gz linux binary * 0.9.9c The 0.9.9.c > /tar/malete.0.9.9c.tgz sources (~ 180 KB) adds > Protocol a terms method , > IIF IIF/ISO2709 import/export and some > CmdLine documentation . Also available as > /tar/malete099c.zip windows binary or statically linked linux binary (diet) > /tar/malete.0.9.9c.gz standard or for > /tar/malete.0.9.9c-cdsunix.gz "UNIX" format CDS/ISIS (each about 47 KB compressed, 86 uncompressed). The windoews binary is known to have a couple of problems on XP, which is pretty incompatible with 98. I will try to fix that once somebody donates me a XP box. * 0.9.9b The > /tar/malete.0.9.9b.tgz preview release adds import and export from/to traditional > CDS /ISIS file formats, both record and index data. * 0.9.9a Finally we have something like a > /tar/malete.0.9.9a.tgz testing preview of OpenIsis 1.0 It comes complete with a server (which even runs on windows!) and a pure PHP library supporting read, write, query and index. There also is a cds test database, consisting of two text files: The mrd (record data) is used as is, and the mqt (query text) is loaded into a mqd (query data as platform independent B-Tree leaves). Testing so far was not all too thorough, however, all basic requests as used in demo.php work as described in > Protocol (TIP: set $db->srv->dbg to true to see the protocol). The protocol is so ridiculously simple that it can be implemented in any language easily. To be more explicit: both major interfaces of any future Malete release, that is the file formats and the network protocol, are FIXED in that what is working as documented now will continue to work the same way in any future release. ("Will" meaning "should", else it's a bug). Main missing issues to 1.0: - file converter from and to WinISIS/DOS ISIS (done in 0.9.9b) - a TERMS method (done in 0.9.9c) - other languages - implementation of filtering as described in > Query The first two are not a big problem and can be expected soon. --- $Id: DownLoad.txt,v 1.16 2005/05/11 08:24:29 kripke Exp $ doc/FileFormats.txt0100644000076400001440000004053410145115777013443 0ustar gabiusers;man malete 5 $Date: 2004/11/12 11:18:23 $ Malete file formats * meta data The Malete options (record 0) file .m0d (with a zero; .mod used otherwise), if present, contains a single serialized record containing metadata, including > RecStruct field definitions Named > CharSet collation info is compiled on demand to byte-order dependent .mcx files with magic 'mcx' or 'MCX' for little-/big-endian, resp. * record data The Malete record data file .mrd, a.k.a masterfile, is a plain text file. Lines are terminated by the ASCII newline character with byte value 10. Any superset of ASCII - including UTF-8 - may be used as character encoding. In other words, byte values 0-127 are always interpreted according to ASCII, values 128-255 have no special meaning. More precisely, the byte values 10 (newline), 9 (tabulator), 45 (minus '-'), 48-57 (digits '0'-'9'), 64 (at-sign '@') and 87 ('W') have structural significance. Logically, the masterfile is a sequence of records, where every record is terminated by an empty line. A record is substructured as a sequence of non-empty lines, each terminated by a newline. Field lines are substructured as a number (optional minus followed by digits) followed by a tabulator followed by any characters but a newline as field values. If the first line of a record does not start with a number, it is used as header line. All other lines must be field lines. As of now, there is only one valid header line format defined, consisting of the letter 'W', followed by a tabulator and a data record header. The data record header is of the form 'rid[@pos][*TAB*leader]'. Here - rid is a positive record id (a.k.a. masterfile number MFN, in ASCII digits) - if @pos is present, pos must be the position of the previous version of this record as byte offset from beginning of file in ASCII digits. Malete software supports retrieval of old versions based on this. - the optional leader is arbitrary leader data for the record. It can hold a MARC leader or a unique key ("name") for the record. If a record has no header line, its record id is one plus the highest record id used up to that point in the file. Using header lines, records can represent updated versions of previously stored records. There is no way to delete records, the operation next to it is to write an empty record. During normal operation, data in the masterfile is never updated, all writes are appends. Thus live backups and replications can be based on 'tail -f' writing to a tape or piped through netcat. A consistent snapshot can be made by recording the current file size and only accessing record versions at lower positions. Compacting a masterfile by purging old versions is straightforward, but considered an offline operation (basically printing current versions to a new masterfile). Compacting a mounted database is feasible for the exclusive mode server, but currently not supported. The masterfile is a special case of the serialized > Protocol , see there for details. The suggested handling of malformed records should not be relied upon. * record access The Malete record access file .mrx, a.k.a cross reference (xref) file, is a binary file dependent on the platform byte order and pagesize. Since the xref file is accessed by memory mapping, its size must be a multiple of the pagesize (which is 4 KB for most Intel based systems). Where the xref is found to be missing, have the wrong byte order or size or be otherwise invalid, it is rebuilt from the masterfile. The xref is organized in units of a fixed size ranging from 8 to 16 bytes. The unit at offset rid*size is a pointer to the current version of record with id rid in the masterfile (rid starting from 1). This pointer consists of - 4 to 8 bytes for the record's position (including any header line) - 3 to 6 bytes for the record's length (including the terminating empty line) - 0 to 2 bytes for the record's number of fields (including 1 header field, even if it's empty) When not compiled with support for large files, only the combinations 4,3,1 and 4,4,0 are implemented. If a record's number of fields exceeds the representable range, the corresponding pointer bytes (if any) are set to 0 and the number of field must be determined while reading the record. This number is also 0 for "deleted" records (no fields, empty leader). The first such unit (at mrx offset 0) consists of - 3 bytes magic number, which is "MRX" for big endian, "mrx" else - 1 byte type, computed as (bytes for pos-4)*16+(bytes for length-3)*4+(bytes for fields) - a 4 byte integer in native byte order holding the max used rid - if the unit size is 10 or greater (i.e. with largefile support), the following 2 (for 10,11) or 4 bytes are an integer in native byte order holding high order bytes of the max record id. * query data and access The malete query data .mqd and query access .mqx files hold the leaves and forks (inner nodes) of a B-Link-Tree, resp., which is usually used as index associating keys generated from masterfile records with pointers to such records. Both files are organized in fixed size blocks containing binary integer numbers and arbitrary key strings (encoded according to the collation configuration). Blocks contain - 16 bytes of "header" data - a "dictionary", which is an array of 4 byte units describing entries in the block with position, length of key and number of values. - a "stack" holding the entries, growing downwards from the end of the block. For dictionary slots d[0]..d[n], describing entries of size s0..sn, entry i occupies si consecutive bytes starting at (block size)-(s0+...+si). The header contains 8 unsigned binary numbers: - num 4 bytes block number - typ 1 byte block type (bitmask; see below) - ksz 1 byte maximum key length (default 0 treated like the maximum 255; CDS/ISIS uses 30) - ptr 1 byte pointer type (bitmask; see below) - lev 1 byte level over bottom (0 for leaves) - nxt 4 bytes number of right sibling block (0 if none) - ent 2 bytes number of entries (length of dictionary) - stk 2 bytes offset of 1st byte used by the stack The first 5 numbers are actually redundant since the block number must match the blocks file position, the three configurable bytes must be the same among all leaves and forks, resp., and the level is not really needed. However, wasting this 8 bytes serves as a minimal check and makes handling much easier. In leaf blocks, the byte order of all header numbers is little endian (so big endian machines have to swap bytes when reading/writing leaf blocks). In fork blocks, the header numbers are in native byte order (since they are usually not accessed by explicit reading and writing, but through memory mapping). The block type is: - upper 2 bits 0xC0 give the basic block type: 0x00 for a standard leaf block, 0x40 and 0x80 for little and big endian forks, 0xC0 for leaves with unstructured values. - next two bits are clear and reserved for future extensions; should software see such a bit set, it must not assume anything about the index structure. - the bit 0x08, if set, will indicate that the index is compressed. Each key is stored as one byte giving the length of the common prefix to its predecessor followed by changed bytes. This is not yet supported and current software will refuse to access such an index. Also do not confuse this with compressing each key individually based on > CharSet collation recoding - lowest 3 bits 0x07 give the block size. For leaves, this is freely configurable as bitcount-9, from 0 for 512 (2^9) bits to 4 for 8KB (2^13). For forks, this is bitcount-12, from 0 for 4KB (2^12) to 4 for 64KB(2^16), and must match the system's pagesize (should there be a system with a pagesize of less than 4 KB, 4KB is used). The pointer type describes the structure of leaf values: - upper two bits 0xC0 give the number of bytes to hold the tag from 0 (0x00) to 2 (0x80). - next two bits 0x30 give the number of bytes to hold the record id - 3, from 0x00 for 3 bytes to 0x30 for 6 bytes - the bit 0x08, if set, indicates that the tag is stored after the record id (as in CDS/ISIS), else the tag bytes precede the record id - lowest 3 bits 0x07 give the number of bytes to hold the position (occurence*65536 + word position) from 0x00 for 0 bytes to 0x04 for 4 bytes. For a value of 5 or 6, we should assume 4 byte position and one or two additional bytes for the record id (currently not supported). Pointers must use at least 4 and at most 14 (2+8+4; currently 12=2+6+4) bytes. Pointer type 0x8B (3 byte rid + 2 byte tag + 3 byte pos) is the same as used by CDS/ISIS. All numbers in the pointer are stored in big endian byte order (most significant byte first) for lexical sorting. The dictionary contains 4 byte units, describing the position pos of an entry, its number of values vln and the length of its key kln, which is stored in the 4th byte (0 here is actually the empty key, which is always the first in the leftmost block of every fork level). In a fork block, the first 2 bytes store the pos in native byte order, and the 3rd byte is vln. In a leaf block (max size 8KB), the pos has 13bit and the vln 11bit. The first 3 bytes in a dictionary unit are, independent of platform, 0: pos mod 256 (lower 8 bits), 1: pos div 256 (higher 5 bits) + 32 * (vln div 256) (higher 3 bits) and 2: vln mod 256 (lower 8 bits). Entries in the stack always start at pos with kln bytes holding the key (as the actual key bytes, or, in a future version, compressed). In a leaf block, after the key there are vln values sorted in increasing order as of memcmp. Values have fixed size and structure as described by the pointer type (typically 8 byte each). Should the leaf block type be unstructured values, ptr is the actual value size and no assumptions are made about the structure of values (to be used independent of a masterfile as general purpose B-Tree). A special value of all 0 bytes is used for stopwords; no other value will be associated with the key. For a tag-first pointer type (bit 0x08 clear), a pointer to tag 0 will be the first and is reserved to store unique keys: at most one pointer to tag 0 will be associated with a key. A leaf key currently always has at least one value; should this key-value association be deleted, the entry is deleted completely. (With index compression vln 0 will denote a stopword). Should a key be associated with more values than fit within a block, the following block starts with an entry with the same key and next higher value. (With index compression, we might consider using the empty key there; to be defined). In a fork block, currently (no index compression), only vln 0 and 1 are used. With vln 0, the key is directly followed by a 4 byte native child block number (which is a leaf block number, if the block's level is 1, else a fork block number). With vln not 0, they key is followed by vln pairs of size (value size + 4), containing value bytes as in the leaves followed by the 4 byte child block number. This is used where a key spans leave blocks: we have a fork entry with vln 0 pointing to the key's first block, followed by an entry with the same key plus the starting value of the next block and so on. (With index compression, we will use one entry with multiple values). Fork block number 0 is always the root, and leaf block number 0 is always the leftmost leaf. All keys and values can be looped in order by starting from leaf 0 and following the nxt pointers. Note that the layout of leaf blocks is fully platform independent: record pointers are organized big endian as in CDS/ISIS, header numbers are little endian, and the dictionary is defined per byte. * shared B-L-Tree access A process accessing the index in exclusive mode may actually shift keys and values around according to the specified layout as it seems fit to minimize unused space in blocks. With shared access, changes are very limited: - Deleting a key-value association is done in the leaf block only, there are no updates to any fork or other key block. - The same holds where an inserted key-value association fits within the target leaf block. - The only case where data is moved between blocks is when a new pair does not fit in its target block: a new block is allocated to become the new right sibling of the target block and as many entries are moved from the block's end to its new sibling so that the new pair can be inserted in one of the blocks. On such a block split, a new entry is made in the block's parent fork (which might trigger a split there). - Should a process find that the key expected in a block is greater than the greatest key there, it must assume that a block split occurred and follow the nxt link to inspect the block's successor. (That's why it's called B-Link-Tree; invented by Lehmann and Yao). Actually, as we use full fork file locking, this can only happen in leave blocks. * limits according to file formats The masterfile obviously has no limits but filesize. To break the custom 2GB (32bit signed) barrier there are two approaches, compiling with large file support and splitting files, discussed further below. In general, most 4 and 8 byte numbers are assumed signed entities, since several system calls handle them that way. Any 2^x here is to be understood as 2^x-1. The xref in the small file implementation (8 byte pointers) can handle - any legal (i.e. up to 2GB) masterfile size. This limit can be broken by masterfile splitting. - any record size fitting in there using 4,4,0 pointers or records up to 16MB size using 4,3,1 pointers (which have a slight performance advantage) - any number of fields - record ids up to 2^31 The full xref spec can handle - large file sizes up to 2^63 (about 8.000.000.000.000.000.000) - record sizes up to 2^48 (256 TB) - record ids up to 2^63 The index can deal with - general purpose key-value pairs of a combined length of up to 255 bytes (might be limited to 127 with extensions like index compression) - standard values (hit pointers) of 4 to 14 bytes (default: 8) allow for keys of up to 241 to 251 bytes (default: 247) - pointers to record ids up to 2^63 (currently impl. up to 2^48) - pointers to non-negative tags up to 2^16 - pointers to position information up to 2^31 (by convention used as 1 or 2 bytes for field occurence, last 2 bytes for word position). - up to 2^32 leaf blocks of up to 8KB, totalling to a leaf file size of up to 32 TB (with large file support). Assuming an average key length of 16 and on average 10 values of 8 bytes per key, each block can hold 81 keys, totalling about 320 billion keys. The fork file, on an IA64 configured with pagesize 64K, can extend to 256TB. This limit can be broken by index splitting. * limits of the current implementation Records have to fit into available memory, which, even when using ridiculous amounts of RAM and/or large swapfiles, is bound by addressable memory. On 32 bit architectures, this is usually 1 or 2 GB (on the heap). Also bound by addressable memory is the possibility to memory map the access files. While they work without memory mapping, performance degrades substantially. So, to use very large databases, the system should be compiled for a 64 bit machine, which are luckily becoming affordable these days and we hope to get us such a box in the near future. * extending file size limits Neither large file support nor file splitting are currently implemented. However, large file support is pretty straightforward. File splitting for masterfiles works by configuring a number n and using a series of masterfiles f0, f1 ... so that records with ids in the range i*n... (i+1)*n-1 are stored in masterfile i. Likewise a series of xref files is used based on a number m, which should be some multiple of n. File splitting for the B-Tree is based on configuring a sequence of keys k0=empty, k1 ... and using a series of leaf files l0, l1 ... so that keys less than ki and not less than k(i-1) are stored in li. Fork files could be split on the same keys or some subsequence of those. The advantage of file splitting over large file support is that it is more portable, saves some bytes on file positions and could aid backup, especially where records are mostly appended to the last masterfile. Moreover it can extend indexes even beyond 32 TB. The total database size need not even be addressable with 64 bits. The disadvantage is that it is a little bit more complicated and when used exhaustively could make the system's open files limit become a problem. Also tracking and snapshotting masterfiles is a little bit less trivial. --- $Id: FileFormats.txt,v 1.7 2004/11/12 11:18:23 kripke Exp $ doc/IIF.txt0100644000076400001440000001704210124533404011621 0ustar gabiusersIIF, MARC and Z39.50 IIF is the "Information Interchange Format", a record serialization format specified in ISO standard 2709, also published as ANSI > http://www.niso.org/standards/resources/Z39-2.pdf Z39.2. IIF is mostly a plaintext format, in that almost any information is encoded using ASCII characters (no binary numbers) and the only control characters used are byte values 29 (record terminator RT), 30 (field terminator FT) and 31 (as subfield delimiter). > http://www.loc.gov/marc/ MARC ("MAchine Readable Catalogue") is actually a family of largely incompatible standards ( > http://www.loc.gov/marc/marcdocz.html USMARC , > http://www.ifla.org/VI/3/p1996-1/sec-uni.htm UNIMARC , UKMARC, ...) that evolved from MARC I (1965). While the main concern of the MARC standards is to specify actual data models (assigning tags and subfield codes, which can be used perfectly well in Malete, CDS/ISIS or other databases), they also specify a variant of IIF as suggested common format for data exchange, which we here refer to as "MARC". (This file syntax seems to be mostly the same for all MARC standards). > ftp://ftp.loc.gov/pub/z3950/official/part1.txt Z39.50 is a network protocol to search and retrieve records. It supports various query "languages", the most commonly used of which is called Type-1 query. Type-1 is similar to the queries as supported by Malete and CDS/ISIS, however, much more general and complex. Terms can be searched for in any indexed field or with restriction to one or more "attributes". Attributes are basically the tags used in the index, which are almost always different from those used in records. While it is common for records to use any of the various MARCs or even completely different formats, the attributes used in bibliographical systems are typically those specified by the Bib-1 attribute set (e.g. assigning 4 to title). Z39.50 allows a client to select a record format from various conversions supported by a server. When a MARC format is selected, the data is actually transmitted serialized according to IIF. * IIF and MARC serialized records IIF specifies a serialization for records. Like the Malete record data file, an IIF file is simply a stream of such records; there is no additional file header. A record has - a 24 byte leader, containing 16 bytes structural data and 8 bytes application data (x, imported as "MARC leader"). The format for MARC is LLLLLxxxxx22BBBBBxxx4500. The Ls and Bs are total record length (including leader and a terminating RT) and start of data (field values, after an FT terminating the dictionary). The first '2' denotes that every field starts with two indicator bytes, the second is the subfield identifier length including the delimiter char. - a "dictionary" array with one entry per field containing 3 bytes tag, and n and m bytes for length and offset. n and m are digits at leader offset 20 and 21, MARC uses 4 and 5. In general IIF, leader byte 22 may specify a number of implementation defined entry bytes. - the actual field values, each terminated by the FT character. As opposed to folklore, MARC does NOT use a '$' as subfield delimiter, nor a '#' for unused indicators. Rather, the examples in the specs use a '$' to REPRESENT the subfield delimiter control character 31 (^_), and a '#' to REPRESENT a blank. The RT(29, ^]) is sometimes represented as '\' and the FT(30, ^^) as '^' or '@'. * Malete IIF import and export The malete tool provides two rather simplistic > CmdLine commands iifimp and iifexp. The command specific options are: - Ffile specify full filename for the IIF files. Default is the basename of the Malete database with extension .iif. On UNIX, a filename '-' selects stdin/out. - Nomarc (literally) do not assume the MARC structure 22/450 on import. Requires proper IIF data. - P[iic] on export, prepend indicators ii and, where needed, subfield c. A single -P uses two blanks as indicators and subfield '0'. Suggested to produce at least syntactically correct MARC. - Rid (literally) on import, use a numeric control number (1st field, if it has tag 1) as record id. Note that on export, the record id is always used as control number unless the record already has one, since this is specified as a must not only by MARC, but by IIF. * creating proper IIF from WinIsis In Database-Export, set the subfield separator to \031 and output line length to 0. If the fields do not contain valid MARC data, use a reformatting FST like $ 001 0 MFN 044 0 |00^a|,v44 024 0 |00^a|,v24 026 0 |00|,v26 070 0 (|00^a|,v70/) $ Make sure, that - the first output field is tag 1 containing some unique id - every field starts with two indicator characters (really should be blank, but that would be stripped during export) - the indicators are followed by a delimiter and subfield identifier Still the output is not 100% correct, since WinIsis sets number of indicators and identifier length to 0, where MARC specifies 2. However, many other MARC processors, including zebraidx, ignore these settings. * making MARC data available via Z39.50 MARC records can be made easily available using indexdata's > http://www.indexdata.dk/zebra/ zebra. If records in your IIF file use tags and subfields conforming to, say, USmarc, simply check out the test/usmarc example in the zebra distribution. Put your data in the records subdir and run "zebraidx update records; zebrasrv". If your data was exported from WinIsis, you may want to put a line "encoding Cp850" in the .abs file. You must use recordType: grs.marc.something, meaning that it's general structured data in some marc file format. The sample usmarc.abs uses the "marc usmarc.mar" statement, and usmarc.mar (in the zebra/tab directory) contains "reference USmarc", stating that the marc input actually IS in USmarc. This need not be the truth, it just means that the records will be served as is, if a client asks for USmarc. However, only the tags listed in "elm" statements in the .abs files will be indexed. Note that zebra's indexing support is not as flexible as that of CDS/ISIS: you can only select fields or subfields to be indexed in one of a couple of modes (like word or phrase). To take full advantage of sophisticated CDS/ISIS FSTs, include them in your export reformatting FST. Use some otherwise unused field tags to hold the index terms and "elm" statements to map them to bib-1 attributes. Omit those fields from the display mapping. To keep the data in its native format (say CDS), change the elm statements to map the fields to index to the corresponding bib-1 attributes for searching, e.g. "elm 024 Conference-name !", and, instead of using the "marc usmarc.mar" statement, create one or more maptabs to map the full record to one or more USmarc a/o other presentation formats as applicable. Check out the gils-usmarc.map example in the zebra/tab directory. Consult the > http://www.indexdata.dk/zebra/doc/ zebra documentation for details. * links - ISO2709 "Information Interchange Format", a.k.a. ANSI/NISO > http://www.niso.org/standards/resources/Z39-2.pdf Z39.2 - Machine Readable Catalogues > http://www.loc.gov/marc/specifications/specrecstruc.html (US) MARC 21 , > http://lcweb.loc.gov/marc/ overview , references at the > http://www.oasis-open.org/cover/marc.html Cover Pages - Z39.50 > ftp://ftp.loc.gov/pub/z3950/official/ official spec , overview at > http://www.oclcpica.org/content/45/pdf/z3950_handbook_paper.pdf OCLC|Pica , links at > http://www.indexdata.dk/technologies/z3950/ indexdata , makers of excellent free Z39.50 software. - Uncle Aung's > http://uncleaung.com/zisis/ Zisis --- $Id: IIF.txt,v 1.5 2004/09/23 11:44:04 kripke Exp $ doc/MOM.txt0100644000076400001440000002156310045441544011652 0ustar gabiusersAbout cases and trunks: La Maleta and the Malete Object Model. * DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT This document describes two data structures: - la maleta (suitcase or Malete Array MA) is a flexible and lightweight two dimensional array, which can be represented (stored, exchanged) as and provides an interface to a Malete Record - el maletero (car boot, trunk or Malete Object MO) is an extended maleta, supporting a DOM-style tree of contained "objects". The term "object" here, like in the somewhat mislabeled DOM, relates to structure, not to behaviour (methods). * la maleta - the Malete Array While the actual implementation of a maleta (e.g. by means of an actual two dimensional array) is not part of this specification, the concepts are probably easiest to understand by thinking in terms of a Malete Record as described in > RecStruct . In the first dimension, there is a list of fields (pairs of tags and values). Every fields value is typically structured into subfields. The first (index 0) field's value (header) is considered special, it typically contains some record's id a/o control information. Other fields (body) constitute the record's contents. A maleta is considered to be a field (it's field 0) augmented by a body. Like any array-like data structure, the maleta uses an index expression to address it's parts for either reading them or assigning to them. Only values can be assigned to; tags can only be inserted or deleted. Like the PHP array, it has a builtin cursor (in the first dimension) for the concept of a current field. Like the Perl array, it supports slices (addressing multiple parts at once) in both dimensions. Here we describe the textual representation of an index, which implementations will typically parse into an internal representation. The parts of an index are, optional, but in this order: - a field spec selecting one or more fields by tag or position - a subfield spec selecting one or more subfields by id or position - a range spec selecting an offset and length - a key spec selecting a key to match Every part has an operator and value (id). An index may address multiple fields or subfields. Selecting both depends on the implementation supporting nested lists. Implementations may ignore whitespace in index expressions. The field spec uses a numerical value as tag or position. Addressing a single field sets the cursor to that field: - '-' first: reset to head and move to next (having tag=id, if given). - '+' next: move to the next element (having tag=id, if given) without resetting - (none) current: no change with no id or if cursor is on tag=id, else first. - '@' index: selects the ith element, using id as index (0 is head). Addressing multiple fields: - '--' loop: loop elements having id, returning a list of the individual results. Without an id, the list contains alternating tags and values. - '++' end: loop at end; useful to append fields - '@@' values: loops, returning a list of values. The subfield spec defaults to none, selecting the entire field value. - '^' subfield: selects the value (with id cut) of the subfield with this id. Id may also be the pseudo subfield '&', selecting the tag, or '@', selecting the cursor position. - '?' test: returns boolean 0/1 whether the field a/o subfield (with id) exists. - '!' break: returns the field or (with id) subfield value, breaks processing else. - '#' position: with a number, selects the ith subfield value, including any id. Addressing multiple subfields: - '^^' subfields: returns a list of subfield values (with id cut) for the given id or all. Without an id, the list contains alternating ids and values. - '##' position: returns a list of unmodified values (i.e. a split on subfield delimiter). A range spec can have one or both, in that order, of the following: - '*' offset: cuts the first offset bytes (not characters) - '.' length: cuts to the first length bytes A keyspec is part of setting the cursor, doing a next while the selected data does not match the specified key. When used with a test or break, it applies to the data (not the boolean result), and, with empty field spec, returns false or breaks, instead of moving to next. - '==' exact: checks for exact match - '=%' prefix: checks for prefix match - '=:' contains: checks for substring - '=~' expr: evaluate key as regular expression (optional extension) Index expressions are independent of any metadata. Especially they do not know anything about fixed subfields, but only check for the delimiter character. Fixed subfields may be accessed using ranges. However, a helper procedure can be set to rewrite bad expressions, e.g. turning field and subfield names into tags, subfield identifiers and ranges. Minimal implementation requirements: - tags may be limited to the range 0 to 65534, inclusive - position ('#'), offset ('*') and length ('.') may be limited to the range 0 to 255, inclusive * array operators Basic operations on maletas are - getting a single index returns the value or list (empty value or list if not found) - getting multiple indexes A failing test or break stops processing. A positive test does not produce an output value. Returns a list of the values returned by each index (unless there is only one non-test index). In Tcl, get is the default operation. Examples, assuming a maleta called v: $ v 24 ;# select the current (or first) field 24 v 24^a ^b ;# list of a and b subfield of current field 24 foreach {i v} [v ^^] { puts "$i=$v" } ;# list all subfields of current v --24 ;# list of all 24 values v td^width ;# helper should rewrite this to 100^w v -24?a:foo .2 ;# the MARC indicators of first 24 field where ^a contains foo $v->get("-24?a:foo", ".2"); # more verbose in PHP, Perl v.get("-24?a:foo .2"); // no varargs in Java, split at blanks $ Assignment (set), like retrieval, takes any number of string parameters. Implementations should also support passing multiple values in one parameter as a list, maleta or serialized record. Depending on the environment, this may require a different or overloaded method. An index addressing a single value (i.e. not a test) takes the next parameter as new value. If the addressed item does not exist, it is created. Assigning no value (there is no next parameter) deletes an item. If multiple items are addressed, all following parameters (or the elements of a single list parameter) are applied in turn. Excess parameters create new items, lacking parameters delete items. Tcl uses a '=' parameter as assignment operator, '=@' to assign from a list. Examples: $ v ^a = $a ^b = $b ;# set current subfields a and b to the variables v --24 = foo bar baz ;# rec has now exactly 3 24 fields v --24 =@ {foo bar baz} ;# same $v->set("--24", "foo", "bar", "baz"); $ Insertion is a variant of assignment addressing newly created items. * el maletero - the Malete Object A maletero (or MO) is a maleta where every field is itself a maletero, i.e. can have a body. It's body fields are called childs. A maletero corresponds to a region (contigous sequence of fields) in a plain Malete record by means of counted or delimited structures. Maleteros come in three flavours: - list (plain vanilla): The maletero behaves exactly like a maleta. All childs are treated as simple fields, regardless of their tag. The MO maps one-to-one to it's record. This is the most efficient mode where no complex childs are needed. - struct (+ strawberry, chocolate): Childs with non-negative tags are treated as simple. A child with a negative tag -n corresponds to a region spanning n fields. This includes one field for the child's tag and header and any fields it's childs correspond to in turn. When looping or setting the cursor, counted subrecords are recognized and their body is skipped over. - mom (with fruit and liquor): in this DOM-style mode, only fields with tag 0 are simple (textnodes). Every child with a positive tag orresponds to a delimited structure. An implementation may or may not support counted structures in mom mode. A maletero provides object handles to it's parent and childs, either by modifying the current handle or by creating new handles. New handles can be based on a copy of the corresponding record or use region in the same record. The latter may not be supported by all implementations or make the objects immutable to avoid conflicting concurrent modifications. * implementations A basic implementation may provide only a maleta, which is sufficient for traditional CDS/ISIS style record access. A complete implementation may provide only a maletero which can be used as maleta (like in english trunk means both car boot and suitcase). A particularly efficient implementation may provide both separately. The initial implementation is a Tcl extension (written in C), optionally augmented by a Tcl module (written in Tcl). The abstract model, however, can be similarly implemented in other languages. --- $Id: MOM.txt,v 1.3 2004/05/03 13:04:36 kripke Exp $ doc/MetaData.txt0100644000076400001440000000211310101201661012653 0ustar gabiusersthe Malete metadata record * overview Every database may have a metadata record assigned. The system may also have a default metadata record containing defaults. The core defines metadata fields 001-009, to be used in the same way by all applications and environments. - 001 name and options as used by the core on the > CmdLine command line - 002 unused (see > TagUse ) - 003 content-type contains a MIME charset and other information mostly for the client for proper interpretation of the data - 004 collation definition or reference used by the core as detailled in > CharSet charsets and collation - 005 version last modified time of this metadata - 006 field definition as of > RecStruct - 007 unused (index definition?) - 008 unused (references?) - 009 unused (stopwords? any?) We may use an CDS/ISIS compatibility block, e.g. 01x, to store the original contents of > http://www.cindoc.csic.es/isis/ap-f.htm files like the .FST or ISISAC.TAB from imported CDS databases for use by some compatibility application. --- $Id: MetaData.txt,v 1.2 2004/07/26 13:14:25 kripke Exp $ doc/MultiProcess.txt0100644000076400001440000004010310035266437013647 0ustar gabiusersConcurrency and locking in the Malete core. * MP vs. MT Unlike the OpenIsis versions up to 0.9, which where built for multithreading, the Malete core is designed to run in multiple processes in parallel. The major reason for this shift is ongoing problems with several MT environments: - multithreading is in a big move towards compiler supported thread local storage (TLS). TLS routines like pthread_getspecific or TlsGetValue are replaced by the __thread attribute on variables. While this is nice and efficient, it is incompatible (it does change the behaviour of libc in subtle ways). - on Linux, the 2.2/2.4 LinuxThreads are now superseded by NPTL in the 2.6 kernels (and some backported 2.4.20s, watch out!), which uses compiler supported TLS. While this offers a couple of advantages, it is not worth the effort to support both during the transition. - on BSD thread support is only emerging. On MacOS X, the implementation has yet to prove its stability. Even on Solaris with a long track of reliable MT support, they are switching the threading model. - on Windows the 9x and Me versions are still in widespread use, which lack important features to leverage the benefits of MT (OVERLAPPED_IO and SignalObjectAndWait). While they also lack support for synchronization of multiple read/write processes, at least read-only multiprocessing is feasible. - while Java can be considered by far the most stable environment for MT programming, this only works out when actually writing pure Java. Embedding MT C-code via JNI is not well defined and pretty intricate, especially where the underlying MT implementation is on the move. On the other hand, there is some demand for MP support in certain environments like e.g. MP PHP or a parallel server running from wrappers like tcpserver. Thus Malete by now focuses on MP and leaves a reworked MT version for better times with stable and widely available TLS support. * shared ressources The only ressources shared are (regions of) files, i.e. there is no shared memory (besides mmapped files), queues, pipes, sockets or other ressources. In typical usage read access is much more frequent than writing. Therefore the means of coordinating access to shared ressources - must support multiple processes - should support read/write locks (i.e. shared vs. exclusive locking modes) to allow for concurrent readers - should support locking and unlocking of regions of open files to support concurrent writers With some limitations to be discussed further below, this can be implementated based on file locking. Features we do NOT need: - locking over network file systems means looking for trouble. While this MAY work in simple cases, it is NOT RECOMMENDED! If you really can't avoid accessing remote database files via NFS or SMB, the only reasonable use of locks is to "protect" read-only access. DO NOT even consider writing your valuable data to remote storage. Better run a database server where the disks are. - mandatory locking means looking for trouble, c.f. /usr/src/linux/Documentation/mandatory.txt. DO NOT mount with mandatory locking support, DO NOT set the mandatory locking file mode! - deadlock detection. The usage of locking is deadlock free. (Unless mandatory locking is enabled, so don't do that). * concurrency modes There are three modes of concurrency to distinguish: - read-only mode: any number of processes holding shared locks on whole files. Every process may read at any time, no process may write. This is the most efficient mode for high volume query processing. - exclusive mode: a single process holding exclusive locks on whole files. The process may or may not write to the files. A simple, safe and portable mode to allow writing. - shared mode: multiple processes using temporary locks on regions of files. Every process may read and write. Supported on UNIX-style systems only. The first two modes using locks held during process lifetime are trivially correct. Actually older OpenIsis versions used such locks. Shared mode is much more complicated and deserves more detailled inspection, which is done in the remainder of this document. * shared mode First of all: try to avoid it. For development and typical data entry use, an exclusive mode single process server will do perfectly well. For high volume query processing, use a separate copy of the database in read-only mode. Malete databases are designed to be easily copied and backed up. This is intrinsically much more efficient and reliable than combining high read and write load. Moreover, cleanup tasks like data and tree compaction can be safely done in an exclusive writer, but would require unduly synchronization effort in shared mode. Having that said, here are the gory details. Both the records (r) and the index entries (q) use two files each: - a "data" (d) file (the plain masterfile .mrd and the BTree leaves chain .mqd, resp.) - an "access" (x) file for faster lookup (the record xref file .mrx and BTree inner nodes .mqx, resp.) The access and data files need to be in sync, thus writing access needs to be synchronized to some extend. Typically, but not necessarily, the application uses index entries in turn as pointers for the records; see below. Basically, all files are accessed using (unbuffered) native file IO. The access files, however, are memory mapped where possible. In summary, we use - one "record lock" per database guarding any record write - a "xref lock" for every record - one "tree lock" per database guarding BTree inner node access - a "leaf lock" for every BTree leaf block To read a record: - obtain the shared xref lock - look up the xref - release the xref lock - read the record (does not need a lock since records are immutable) Writing a new or changed record is done by: - acquiring the exclusive record lock - appending to the end of the masterfile - obtain the exclusive xref lock - writing to the xref file (using msync where mmapped) - release the xref lock - release the record lock When searching the index - the shared tree lock is acquired, the tree is read to find the leaf number, and the tree lock is released - a shared lock on the leaf block is acquired and the leaf is read. The shared lock is released immediatly after reading. - on split detection (Lehmann/Yao) successive leaf blocks are read alike The steps to write an index entry are: - find the target leaf block, searching as above, but using exclusive leaf locks. On split detection, an ex leaf lock is released only after locking and reading a successor. The final ex lock is held until after the leaf block has been written. - if the write involves a split, lock the block after the end of the leaves file and double check you really got a brand new block (it must not be readable, else, repeat) - write the block or blocks - if the write involved a split, obtain the exclusive tree lock. Iteratively insert pointers to any newly created blocks. - release all locks To support a unique key, the record lock is held while writing / looking up the record's key in the index and until after the record was written. * operating system considerations Locking an mmapped tree file may not work on some systems. The rationale is that accesses to mapped memory can not be checked against arbitrary mandatory locked regions; c.f. /usr/src/linux/Documentation/mandatory.txt. On some systems locking mapped files is completely ruled out, others, like Linux, deny only mandatory locks. Solaris allows locks on the whole file only (not regions). Since all locks are used advisory, however, there is no need to put the lock on the actual bytes affected by an operation. - The record lock is on byte 0 of the masterfile .mrd, and the xref on record id n (1..) locks byte n. - The tree lock is on byte 1 of the leaves file .mqd, and the leaf block n (0..) locks byte 2*n. Consequently, in read-only and exclusive mode, full locks on the data files .m?d are sufficient to prevent conflicts with other writers. While this statement with regard to read-only and exclusive mode processes can be considered to constitute an interface, the details of shared mode coordination may change (i.e. which bytes are locked in which order, see considerations below). Since M$ Windows does not offer support for serious programming, we are limited to the exclusive and read-only modes: The 9x/Me family is even lacking shared file region locks and locks that can be waited upon, not to mention any reasonable means of signaling. Memory mappings here are of limited use since they might copy the file to swap. The NT-based versions have a couple of the necessary calls like LockFileEx; Still, all this is pretty tedious and the semantics of memory mappings (CreateFileMapping) in the presence of writers is problematic at best. * optimizations to consider - where accessing a memory mapped xref can be considered atomic, xref locks are not needed, meaning readers do not need any lock at all. For 8 byte xrefs, we can easily avoid being interrupted by page faults, so some architectures may support this. - where the effect of writing a leaf block is atomic, i.e. seen completely or not at all by any read, readers do not need to use locks on leaves. According to > http://marc.theaimsgroup.com/?l=linux-kernel&m=107375454908544 Linus this may hold only under very specific conditions: We must have neither SMP nor the new kernel preemption enabled, must have ext2 or similar filesystem, the file region must fit one cache page and the userspace region must not pagefault (i.e. we need a piece of locked memory). We always fit a cache page, if our blocksize is a power of two of at most the cache page size. While 4K is a usual value for the cache page size (as well as the pagesize as well as the filesystem block size), we use 1K (which, under Linux, is the minimum for all of these) leaf blocks to be reasonably safe. - for writers, there is next to no more concurrency we could achieve. The minimum about the changed masterfile structure that needs to be visible is its new size, and basically we are just setting this with the one write operation. With any other scheme along the lines of "first extend, then lock only that area" and so on we would only get a couple of additional syscalls plus integrity issues with much more difficult crash recovery. Especially we don't use a separate lock on the access file. - one might consider delaying a sync-to-disk (fsync/msync) until after the lock is released. However, at least a msync using MS_ASYNC and MS_INVALIDATE should be issued. (On Linux, MS_INVALIDATE does nothing, since maps are always coherent). - locking the whole tree might seem pretty restrictive, however, it saves a lot of syscalls and the operations on the mmap should we very fast. Concurrency is affected only by writes to the tree, which are fairly rare (on about 3% of inserts, depending on sizes and fill level). - for non mmapped tree files, however, the classical per-block locking scheme could be considered to reduce worst case delays. Locks on the odd leaves file bytes are reserved for this purpose. Yet this involves substantial overhead on every read. * considering writer starvation In the presence of readers holding shared locks, we can not expect fcntl to avoid writer starvation. If at every time there is always at least one process having a shared lock, a writer may wait indefinitely for an exclusive lock. This can trivially be avoided by only using exclusive locks, which typically will have a simple and fair FIFO implementation. Clearly this sacrifices most concurrency in a situation where it is obviously demanded. A more sophisticated aproach is to guard the acquisition of a lock A, whether shared or exclusive, by an additional ex "guard" lock Ag in a sequence of get Ag - get A - release Ag. That way no process can get A while another process is waiting for A. This double locking should not be used for leaf reads, because continued overlapping of shared locks on leaves means such a pathological congestion that the system will croak anyway. The record and tree locks, on the other hand, are held by readers only while they are inspecting memory mapped files, which should involve no disk accesses if you are going for high throughput. More precisely, we should only very rarely see a process suspended for any reason while holding such a shared lock, and thus overlapped shared locks should be rare. To summarize, in almost any case of high volume querying it is advisable to set up a separate read-only copy of the database instead of increasing system load by doubling the number of locks. According to these considerations, we might use the aforementioned optimizations to reduce writer delays, but do not care about absolutely avoiding writer starvation. * unmounting databases Whenever a process created a new database, other processes may access it by opening it on demand just as any old database. There is no additional interprocess communication needed to make a database available. Since the shared mode is meant to be used in high volume production environments, we assume any available database to remain available without structural changes, and support such changes only in the exclusive server, where they can be handled internally. For completeness, however, here goes an outline of the steps that would be needed to support unmounting databases in a shared environment. Changes to a database, including making it unavailable, are handled by one process obtaining the exclusive "options" lock. Conceptually this can be regarded as lock on the options file, with a process using the database holding a read lock on the options and a process changing database options requiring a write lock. In such an environment, - any process using a database for read a/o write without changing options holds the shared options lock. - a process that wants to change a database must obtain the exclusive options lock to mount the database in single process mode. - before waiting on the exclusive options lock, the process acquires guard locks and notifies other processes to release their locks For synchronization issues, actually three locks are used: - the "options lock" is held shared or exclusive while the database is in use - the shared or exclusive "use lock" guards any attempt to obtain the options lock - the exclusive "change lock" guards any attempt to obtain an exclusive use and options lock A reader (shared user) - asks for the shared use lock without waiting. - if this fails, another process is attempting modification and the reader must close the database. - else, no process must have the exclusive options lock. The reader obtains a shared options lock and releases the use lock. A writer (about to structurally change the database) - first asks for the exclusive change lock without waiting. - if this fails, another process is attempting modification and the writer must close the database and bail out. - else, no process must have the exclusive use lock. Then the exclusive use lock is acquired with waiting. (Since there are shared use locks held for very short times, this has a slight risk of writer starvation and could, for paranoia's sake, be guarded by yet another lock). - finally other processes are notified and the writer waits for the exclusive options lock (probably using a timeout). - once done with changes, the writer releases locks in reverse order. Notifying other processes: - for the unix multi process server, we have all processes share the same process group, so they can be signalled by kill(0,sig). On most systems we should use SIGWINCH and SIGURG, because they are ignored by default (so we don't kill our tcpserver). - while SIGURG will have any running request aborted, processes block SIGWINCH during normal request processing and receive it only when about to read a new request. To make a database finally unavailable, a process should move or remove the files while holding the exclusive locks. Since locks are obtained on open files, other processes may have opened the files before this and finally succeed in obtaining the options lock. To finally detect this race condition, a process must use stat and fstat to ensure, after getting the options lock, that the file it opened is still the same it would open then. doc/OverView.txt0100644000076400001440000003205210063573361012766 0ustar gabiusersAnnouncing Malete, the database engine powering OpenIsis 1.0 * from 0.9 to 1.0 Based on the 0.9 engine and especially its Tcl binding, we had a system complete enough to do very intensive application testing of all concepts, both handling bibliographical and terminological as well as general industrial data. With those experiences at hand we spent the second half of 2003 to give our then two year old software a complete overhaul, in order to create a basis to last. Along the traditional believes of Unix design we figured out that the best and most stable combination of robustness/performance with flexibility/convenience can be achieved by clearly separating - a general purpose database system which is very simple in order to be fast and robust and lay a solid ground for flexibility, but itself is meant to be accessed by other software (or geeks) rather than humans. While this engine is based on the Z39.2 record model (even supporting record leaders as used by MARC), it makes no special provisions to support bibliographical data or CDS/ISIS legacy, but rather tries to make this model appealing to general purpose database usage. This engine is called Malete (kurdish for "our house"). Malete includes a database core library, generic server and access libraries for various programming languages. - a CDS/ISIS-style application or, actually, like Winisis, a framework for applications. This is targeted at CDS/ISIS users and librarians in general. It provides support for conversion from and to a variety of known file formats including MARC, high level indexing, references (authority files, coded data), forms and so on. In other terms, for retrieval you will rarely need more than the Malete engine (plus some formatting for presentation, which is usually done in a web programming language like PHP), while for data entry you want a convenient graphical user interface providing all sorts of lookups and checks. * technical changes - multiprocessing For a variety of reasons (detailled elsewhere) we postponed support for multi-threading (to at least until after the ongoing move towards compiler supported thread local storage is stable and widely available). Instead writing support by multiple processes is enabled based on file locking. Fast and consistent caching even for processes with very short life time (like CGI scripts) is achieved by replacing the former explicit caching with memory mapping. - platform indepence Now both record and index data file formats are identical across platforms (i.e. the same even on big endians like Suns and Macs). Only the pointer and tree files are plattform dependent, but are rebuilt from the data as needed. - generalized record format A record with n fields is now a series of n+1 tag-value pairs. The tag of the first field is the negative total length -n-1 and the value is a record "header" consisting of the record id (MFN) and optional leader data as used by MARC. Obviously, such a series can be part of a larger record, meaning records can be easily nested. - simplified serialized format The serialized (textual) representation of a record, used both in the masterfile and the server communications protocol, has dropped low-level support for field values containing newlines. Where needed, the application must apply proper encoding (but tools for that are provided). - simple transaction support Updates to a record can optionally be qualified with the position from which the record was last read, having the update fail if the record has been modified meanwhile. Reads can be done in "consistent snapshot" mode, reflecting the state of the database at one given point in time. - unified message interface The server communications protocol has been simplified and straigthened out. The masterfile now is only a special case of this protocol and thus can be directly sent to a server. Conceptually every record is a message saying "write me". - ucspi based server The server is designed to run under tcpserver, meaning it can take advantage of all of its features like access control, basic client authentication, IPv6, SSL encryption and so on. For more details see > Diff09 * applications and components OpenIsis 1.x will provide the following applications and components (probably not all in 1.0, but 1.1 should be fairly complete): - the Malete database server for Linux and other UNIX-like systems written in ISO C. This is aimed towards miminal functionality at maximum performance. Intended usage is for high volume read only processing and read/write with application controlled indexing. On UNIX, the server will be multi process based. On Windows, use of multiple processes is restricted to read-only mode. - Malete and OpenIsis command line tools for all systems providing several tools including conversion from and to legacy CDS/ISIS file formats. - Java, Perl and PHP libraries to contact a server, all written completely in the respective language. These are aimed at tight language integration, leveraging the application language's strengths and programmer's skills. Will run on all systems as supported by each language. - a Tcl extension and library where the library acts similar to those for other languages (but based on a C implemented record) and the extension basically provides the server interface in process. - an application server for all systems (i.e. including Windows), providing database and http service, based on Tcl with or w/o Tk. While this will not achieve the high throughput of a purely C-based server, the Tcl layer can add virtually arbitrary functionality. Intended usage is for read/write with server controlled indexing and integrated http applications based on Tcl server pages. Servers based on other languages are waiting for volunteers. - a Tk based GUI for all systems. Can run standalone or acting as server and/or client. - the OpenIsis Tcl library providing support for CDS/ISIS-style applications, e.g. indexing similar to FSTs. - the OpenIsis application targeted towards users from the CDS/ISIS community, esp. librarians, to provide interoperability with existing ISIS databases and support for bibliographic formats in a user friendly way. Written in Tcl/Tk as a sister of OpenMLCM. * Malete modules The Malete database system is structured in the following modules: - core basic C library for handling, storing and retrieving simple records. - pw "patchwork" framework for high level database services based on message passing. Some designs are borrowed from the Lisp and Smalltalk languages. - tool helper functions and command line tool including communication utilities and standalone server - java, perl and php client modules - tcl extension and base library - app the Tcl based application server - gui a generic Tk graphical user interface On top of this, the OpenIsis 1.x application set contains: - old compatility functions and command line tool - isis the OpenIsis library and graphical user interface * ISAM core This implements a variant of ISAM (index sequential access method) based on the ideas of Z39.2 (IIF) and Z39.50 (Type-1 queries). It provides a fully open and unprotected interface for unrestricted access at maximum performance. The core library is not fully self contained, but will require a few functions like stream I/O to be provided by each environment. It makes only very limited use of metadata, dealing with "physical" aspects like file names, locks and character sets. - util basic list, sessions, output buffers and other utilities - system services like file IO and time - charset recoding and collation - storage set of functions for database file access (master file and b-tree) * patchwork The patchwork C library wraps the ISAM core into an extendible framework for high level database services, based on passing records as request and response messages to server objects. It provides a fully abstract and generic method call interface plus a couple of database objects. An object dispatches messages by checking their type and other parameters and taking appropriate action, including forwarding to parent objects. This is known as the "pure object oriented" approach, as these objects don't have any other interface but the message dispatcher, especially no directly accessible data. - struct higher level operators on ISIS records a la IIF (Z39.2/ISO2709) based on meta data, including various substructures - base dispatcher wrapping the ISAM core. Based on the 0.9 server, but with some modifications to allow for most efficient message passing. - query dispatcher for ISIS/Z39.50 Type-1 style queries - server dispatcher providing record relations, views and other magic * design guidelines requirements: - flexible and efficient buffered pushing of output. Pulling is not used on lower levels; every environment will solicit input on the outermost level as adequate. - flexible and efficient construction, manipulation and passing of records, especially embedded subrecords in the patchwork. principles: - everything is a list. Similar to Java's String and StringBuffer, there is the immutable "Rec" and the mutable "List". - uniform stream output. Conceptually, all output is a list. There is only one (output) "Stream", which may be backed by memory buffers, files or other channels like a GUI window, so even diagnostic output can be captured. - negative counted subrecords. The patchwork uses negative counted embedding, since this allows to pass on embedded records without any modifcation or copying. - low tag usage. Besides reserving all negative tags for embedding, only a minimal amount of tags should be defined. Instead subfielding will be used extensively. The patchwork message header uses tag 0, containing the message type as an indicator, followed by any number of simple options and parameters, resembling a command line (see below). Alphabetic keywords and mnemonics are favoured over numbers. - leader There always has been some out-of-band data on records like their mfn. This is now generalized in the concept of a record leader (see below). implementation notes: - immutable lists are just the same as a record embedded by negative counting, i.e. an array of fields, with the tag of the first being the negative total field count. - record leader The tag of the first field of embedded records contains leader-like meta info; for database records this is (optional) mfn plus a MARC leader. Since there should not be a difference between the representation of embedded and first level records, every record has a leader. - message leader A record representing a message also has a leader. Where the message is not embedded, it is sent as a leading 0-tagged field. Since message leaders start with an alphabetic character, the 0 and tab are omitted in the textual representation. Message leaders use tabs as separators and start with a word indicating the message type to the dispatcher. Following subfields are parameters, with or without identifiers. - getopt command lines a command line of the form "command -aopt1 -bopt2 arg1 arg2" can be easily and canonically wrapped into one field by removing the '-' option indicator and identifying the non-option args as subfield '@'. A commandline interface thus maps easily, and without the need for looking up meta information, to a message leader, from which the method identified by "command" can fetch options using a getopt-like utility. System and db parameters are likewise stored in the options file. - message body most messages use only one type of record parameters; however, special embedded records like indexing instructions can be recognized by their leader, where applicable. Where a message contains parameter fields (first level, not leader subfields), it must use positive tags for that, preferrably using low numbers. - direct embedding Where a message has no parameter fields, i.e. no parameters besides its leader's subfields and embedded records, and there is only one parameter record, the message may, as a convenient shorthand, allow to specify the embedded record's leader (mfn and db for database records) as message options and have its leader immediatly followed by the record data. In other words, the message sort of embedds itself in its parameter record's leader (and has to remove itself before passing it on). This is the form used by masterfile metalines (with ommitted 0). - system options can be specified on the command line or in a system options file. there is a global options list (e.g. verbosity) and per db options (like file paths and readonly). The commandline format is "-aglob1 -bglob2 dbname -xdb1 -ydb2 [... dbname ...]". The system options file contains (the textual representation of a record with 0-tagged) fields, one for each db, wrapped up like "dbname xbd1 ydb2" (with tabs). Those options are NOT stored in each db's .opt file or meta record. - database metadata contained in the db's .m0d file is basically a chained message to the core engine, mostly configuring the "transmission format" > http://uk.travel.yahoo.com/t/wc/germany/berlin/nightlife/malete.html Malete --- $Id: OverView.txt,v 1.4 2004/06/15 13:17:37 kripke Exp $ doc/Protocol.txt0100644000076400001440000004070010230226367013015 0ustar gabiusersThe Malete server protocol. * introduction The Malete server is based on passing of messages, which are represented as records. The only interface to the server can be regarded as a single function "send", which takes a record as parameter and returns a record. The result record itself is a valid message. This "send" can be actually invoked in one of two ways: - by having the server in process i.e. by actually calling the C function "send", possibly via some wrapper to interface another programming language. This is the way the Malete Tcl extension works. - via some bytestream This can be regarded as just one of the wrappers, interfacing a bytestream by deserializing message records from the bytestream and serializing result records to the bytestream. The standard server process uses stdin and stdout and thus can be invoked by executing it from pipes or by contacting it via TCP, when running from > http://openisis.org/Doc/UcspiSsl tcpserver. As a special case, the record data file itself is such a bytestream, however only containing simple write messages. The server maintains a session state bound to a bytestream, e.g. one TCP connection. * messages and data In Malete, every record has a "header", which is the value of the first field. The header specifies which message the record represents, with the following fields ("body") containing parameter data for the message. Recall that - the first field's tag denotes the number of fields in the record - a "data record" is a record that can be written to a database. This requires a record id (MFN), which, however, can be 0 to denote an append with the next available id. - for a data record read from or written to the database, the header will/must be empty or start with a digit. The general format is 'rid[@pos][*TAB*leader]'. Rid is the record id (MFN), which on write may be 0 to append a new record. Pos is the optional old position to guard an updating write against concurrent changes. Leader contains arbitrary data like e.g. a MARC leader, a record key or a message header. Proper message headers are not empty and do not start with a digit. The first token of a message header (up to a *TAB* or end of value) is the message name, optionally qualified by a message target, i.e. an object to receive the message (usually a database). However, messages and data are converted into each other canonically: - If a data record header is encountered where a message is expected, it is treated as a write message as if 'W*TAB*' where prepended (which oviously will write just this record). Even the empty message (a record with 0 fields) is a valid message and will append an empty record when sent to a database. - If a message is treated as data, its header is treated as leader as if '0*TAB*' where prepended. * message targets are objects A server processes messages by first looking up a target object by inspecting and stripping an initial addressing part of the message header (or resorting to some default) and then passing the message to this object. (Actually, even this dispatching is done by an object, the session). In general, objects are free in how they process messages. For example, an object might represent a (session on a) remote server, and simply pass every message there. Objects using the same processing function are said to be in the same "class". Commonly processing functions handle only some known messages and pass anything else on to the function of another class, which is called "inheriting from this class". Objects to which messages can be send are - a structure is a collection of other (child) objects like databases (tables). It does basically nothing but passing messages to its childs. It may support a listing of the known childs. The structure interface may be implemented locally or as a remote server. - a database (table) supports reading and writing of record and query data. A database is a structure, it may support childs e.g. to provide views. - a session is a structure representing the connection to a (local or remote) server. It passes messages to the server's childs (like databases) and maintains some state, called the environment. Any object should recognize - the comment '#' a special message used to pass additional info (echo/error) A structure in addition recognizes - child addressing '.' if the message name starts with a letter and contains a dot '.', everything up to the dot is taken as the name of a child. After stripping the child qualifier, the message is send to the child. With no additional message, the child's existence is tested and returned in a comment. The qualification can contain several dots, which are processed from left. Therefore, 'a.b.c' means to send message 'b.c' to target 'a', which could be for example a remote server, which in turn is expected to somehow dispatch message 'c' to its local child 'b'. The standard messages a database should recognize are - the write message W writing one or more records to a database - the read message R reading records by record id - the query message Q to search the query data (btree index) - the index message X to write index data Standard message and object names always start with an ASCII letter. As a convention, message names should start uppercase and object names lowercase. Every message returns an error comment message in case of error or another message as specified (possibly the empty message). The body of a message (i.e. the fields following the header) may define a fixed or variable number of parameter fields or one or more records, which are in turn, depending on the message, used as message or data records (generally regardless of their contents): - header only: The message is not using any fields or records as parameter. Such messages treat any body as embedded records (see below) specifying one or more chained messages, which are then processed in turn. A possible but currently unused generalization of this is a fixed number of parameter fields. - parameter list: the contents of following fields is interpreted by the message itself. Many messages use only one type of parameter fields and ignore their tags. - embedded records: Each of the records begins with a proper header field, with the tag being its negative length (including the header). A tag of 0 is treated as using all available fields. Should such a tag be positive or specify a length exceeding the number of available fields, the result is undefined, but either an error or treating it as record using all available fields. - immediate record: Some messages also support a short form, where they do not themselves take all of their header, but only chop off some initial part of it, using the remaining message as record. * write The write message takes one of two forms: - short write (immediate record): The header is of the form 'W*TAB*rid[@pos][*TAB*leader]', and the following fields are the body of a record to write. This message writes the record with header 'rid[@pos][*TAB*leader]' and the body as given by the following fields. It returns a short read message with the record id written. - long write (embedded records): The header is a single 'W'. The body contains any number of embedded records. Multiwrite returns a long read message with the record ids written. With an empty body, long write can be used to test the existence and writeability of a database. Note that there is no special support for deleting records; writing empty records has the same effect. * read Like write, the read message takes one of two forms, all returning a long write for the retrieved records: - short read (header only): The header is of the form 'R*TAB*rid[*TAB*count]'. It reads count (default 1) records starting at record rid. A count of 0 reads any records as available and within the read limit. Note that a read of record 0 retrieves the metadata. - long read (parameter list): The header is a single 'R'. The following fields contain one record id each. Note that - the number of records read at once is limited - read might retrieve older versions of records, if the database has a snapshot position set * query The query message is of the form 'Q[*TAB*query]', where query is an expression in the > Query Malete query language. With parameters, the query message creates a new query as the current. With or without parameters, the query message returns an echo of the estimated remaining result set size, followed by a long write containing the next 'r' records from the current query set (subject to a snapshot like read). The query can contain two parts, separated by a '?': - an index based search defining a result set. If it is empty, the search result set is the entire database. - a filter to be applied on record retrieval. If no filter is specified (i.e. no '?'), only record ids are returned. An empty filter selects every record with all fields. Other filters will select records and/or fields. In future versions, one or both parts might be specified as embedded records. By now, however, the query message is header only. Note that - the size of a search result set is limited This limit applies also to any intermediate result, thus the actual set might be much smaller or even empty due to the limit. Some search expressions might allow larger set sizes, especially the empty one does (since no record ids need to be stored). The returned echo contains several numbers: - estimated number of remaining records, including the ones just read. This number may be wrong for a number of reasons, especially it does not account for filtering. However, if it equals the number of returned records, it is safe to assume that there are no more records. This number is the primary echo code, if it is negative, the rest of the echo is some error message. - number of the query, by which it can be referenced. These numbers are per database. - truncation record id. If not 0, this is a record id where the search was truncated due to the result set size limit. Future versions might support transparent continuation after truncation. * terms The terms message has one of the forms - 'T*TAB*from*TAB*to' Selects terms greater or equal the first parameter and less than the second. Where the second parameter is empty, no upper bound is used. - 'T*TAB*prefix' Selects terms with the parameter as prefix. Using a prefix ABC is just a shorthand for from ABC to ABD. - 'T*TAB*from*TAB*to*TAB*tag' Like the first form, but restrict matches to the given tag (number). Terms are returned as a list (record with 0-tagged fields), where each field value is a count of hits of the term, followed by a *TAB* and the term. The list is limited to the result set size. The full index can be looped by using the last returned term as from parameter for the next invocation. When not restricting to a tag, the hit count is just the number of all index entries for the selected terms. This may be higher than the number of matched records, where a term has multiple hits for the same records. With a restriction to a tag, the count is the actual number of records (even where a term has multiple entries for the same record and tag). If the database uses the traditional fulltext index format (the default), tag 0 selects any tag, else tag 0 selects actual tag 0 entries (unique keys). * index The index message 'X' takes a parameter list of data and control fields. Control fields have tag 0 and change the way the data fields are processed. All other fields contain index data. During processing of the message, a position counter is maintained which is incremented by one for every word (in word or split mode), to the next multiple of the field step (default 65536) for every field (1 in word mode), and reset to 0 on tag change. Every control field contains one or more instructions (as always, separated by TABs): - f[pos] sets default (full field) indexing mode where every data field contains one index entry. The position is set to the given or 0 and then incremented to the field step. - w[pos] Like field mode, but incrementing the position by one. - s[pos] Split mode, where each data field is split into words according to collation info. If the index has no collation info, all characters but the well-known ASCII non-letters are assumed to be word characters. - a[pos] set add mode (default) - d[pos] set delete mode: following index entries are deleted. - m[mode] mode 'H' selects traditional conversion of angle brackets: <a[=b]> is replaced by b (or nothing). mode 'P' or none turns this off. - p*pfx* prepend prefix pfx to index entries - r*id* set record id (defaults to the session's last written record) - [+|-]*tag* where tag is a number, stops processing of the field and treats everything after the next *TAB* as data field with *tag*. With a leading + or -, set mode to add or del, resp. Control instructions may also be part of the message header. The index message echoes a count of the index entries made. * comment The comment message '#' is used to augment other messages. It is header only (executing any body) of the form '#*TAB*code[*TAB*message]', where code is a number. A nonnegative code indicates a success, typically some count. A negative code indicates some sort of error (-1..-10) or notification. Message is arbitrary. This message copies itself to the result. * serialization Message can be represented in byte streams according to the following rules: - Field values (including the header) MUST NOT contain a newline character, else the results are undefined. Where an application must be prepared to handle newlines, it must take care of encoding them (see below). - If the message header is empty, no header is printed - else if the message is a regular message (not starting with a digit), the header is printed followed by a newline. - else 'W*TAB*' is printed followed by the header and a newline. - All body fields are printed as the tag followed by a *TAB*, the value and a newline. - A single newline is printed to terminate the message. On deserialization, if a message starts with a number (digit or -sign), this is the tag of the first body field, and an empty header is to be assumed (equivalent to a 'W*TAB*0' append message). For all body fields, the deserialization must be done in the following steps: - take an initial '-' sign and any digits as tag, defaulting to 0 - skip one following *TAB* character - use anything up to a newline as value Consequently, on serialization: - a tag of 0 may and commonly will be omitted - where a value does not start with a TAB, the TAB may be ommited - where a value does not start with a '-', digit or TAB, both a 0 tag and the TAB may be ommited - where values containing newlines are used unencoded, they will in most cases result in following 0 tagged fields However, ommiting the TAB is considered bad style. The record data ("master") file is simply a stream of data record messages, using headerless mode where possible (i.e. appends of leaderless records). Some easy common encodings are suggested to deal with newline characters: - in "field mode", discard newlines by replacing them with spaces or tabs. - in "text mode", newlines are replaced with vertical tabs VT (ASCII 11, ^K). This maybe reversed to restore newline-separated lines if needed, but e.g. on printing the VT will have the desired effect. - in "binary mode", newlines are replaced as VT followed by a byte value 1, if the newline is followed by a byte value 0 or 1, else by a single VT. A VT is replaced by a VT and a 0 byte. - as an "ultra robust binary mode", use BASE64. The advantages of text mode over binary mode are - it is slightly faster than the binary translation - the serialized records do not need more space (whereas the binary serialization might need twice the space) The binary mode has the advantage of not loosing vertical tab characters that might have been contained in the original field values. It is fully transparent and can be used to store any binary data like images with an average overhead of 0.4% (as compared to +33% with BASE64 encoding). Note that for a plain text not containing control characters 0, 1 or 11, text and binary mode have the same results, thus it is reasonably safe for client libraries to use binary mode by default on all communication. However, BASE64 has the advantage of even surviving a character set recoding, thus is more robust for databases which may be exchanged internationally. Also the overhead of BASE64 is fixed to 33% (4 bytes for every 3), while the binary mode has a worst case of +100% (on all VTs). --- $Id: Protocol.txt,v 1.13 2005/01/24 17:51:02 kripke Exp $ doc/Query.txt0100644000076400001440000004675610075746477012362 0ustar gabiusersMalete query expressions: Like CDS/ISIS, Malete supports both index based searching and record based filtering. Unlike in CDS/ISIS, the basic filtering syntax is not based on the formatting language, but a simple extension of the search syntax. However, a more general record manipulation language, the "patchwork", is planned, which will also act as a filter based on complex conditions. In a query expression, everything up to a '?' is a search expression, the remaining part defines a filter. In general, an expression defines a result set, consisting of pointers to occurrences of keys in records. For searching, these pointers are fetched from the "inverted file", the .mqd B-Tree. On filtering, the expression creates pointers as needed from the current record and matches the record, if the result set is not empty. An expression consists of - simple expressions creating result sets - operators modifying result sets - parentheses to control evaluation order * pointers A pointer (to an occurrence of a key in a record) is a 4-tuple (a point in a 4 dimensional space) (r,t,o,p) of - 1 the record's id - 2 the tag of the field where the key was found, so Smith in an author field can be distinguished from Smith in title - 3 the "occurrence" (repeat count) of field, so one can demand Mark and Twain in the same author field, instead of Mark and Twain in two author fields - 4 the position (nth word) within the field * simple expressions A set of results can be created by - [rel][whitespace]term a relation to a term, producing all pointers to keys that have the given relation (as listed below) to term. Relation defaults to equality =. term$ is a legacy form of %term. Term is a series of "word characters", which are ASCII letters and digits, the underscore and all non-ASCII characters (code greater than 127), or any characters in double quotes ". Between the quotes, two quotes evaluate to a single quote. - #n a reference to the nth query. Inserts the search or filter expression, as applicable, of the nth query. (Might not be fully implemented). In a search expression, an existing search result set might be used. As a special case, if the whole query expression is only #n, the nth query is used as is, including search, filter and cursor. Relations are: $ = equality (not needed) % prefix match > greater than >= greater than or equal < less than (not yet implemented, behaves like '>'. For range queries use '-' rather than '<' '*' '>' ) <= less than or equal (not yet implemented, behaves like '>=') : contains (filtering only) ~ regular expression match (optional, filtering only) $ The regexp match will not be available in all environments (since it requires a regexp implementation, probably the one from Tcl). Yet, it's just too tempting ... The less/greater than (or equal) operators are not used alone, but in "key identity" expressions to define ranges. * key identity The semantics of pointers is to denote a word position in an occurrence of a tag in a record. We assume that a pointer is only found at the one key being (some transformation of) the word at that position, i.e. pointers are unique. Therefore, matching arbitrary result sets on pointer identity is of little use. However, when applied to terms, we can use a pointer identity condition as a key identity condition, effectively limiting the range of keys searched: - - key identity is a pseudo operator, syntactically operating on result sets, but actually implemented as joining and modifying term relations. Both operands of - must be terms, it refuses to operate on other expressions. If the left hand term has the = relation, it is modified to >=. If the right hand has the = relation, it is modified to <. The expression 'A - B' will search all keys greater or equal "A" and less than "B". To include "B", use 'A - <=B'. The prefix '%ABC' is actually just a shorthand for 'ABC - ABD'. If multiple upper or lower bounds, including those from a prefix, are found in the operands, the lowest lower and highest upper ones apply. Currently, only the default case greater or equal to A and less than B is implemented. * tag filter The tag filter selects pointers with certain tags by restricting the second dimension to one or more tag values: - expr/tag or expr/(tag[,tag...]) where tags are literal numbers. In other words, it intersects the left hand result set with the set {(r,t,o,p)|t element of taglist} As a special case, a tag filter with an empty left hand expression can be used at the very start of (the outermost level of) an expression: - in filtering to select the fields for final output. This is a proper part of filtering, since a record with no matching fields is skipped. - in searching to force a given ordering. This is not a proper part of searching, but rather defines a cursor for record retrieval, which then uses the search result set as filter. We propably should define an extended syntax for ordering, like we have for filtering, to allow key range conditions here. Since this does a full index scan while retrieving records, results are not made unique, and ordering will be most useful only with a single tag (or some alternative tags) of a non-repeatable field. Currently not implemented. The tag filter, like the pointer identity, is another pseudo operator, actually modifying its subexpressions rather than their result sets. See below for details. * binary operators Additive operator: - + is the logical OR calculating the union of result sets All other operators reduce their left hand result set by selecting elements based on certain relations they have to those of the right hand set. In other words they are intersections of the left hand with a certain set defined (but not actually created) based on the right hand set. Multiplicative operators select pointers from their left hand set based upon their equality to pointers in the right hand set in the first n dimensions: - * ("AND") is 1-dimensional equality selects pointers with the same record id, i.e. those pointer from the left hand set for which there exists a pointer with the same record id in the right hand set - ; or (G) is 2-dimensional equality selecting on same (record and) tag - , or (F) is 3-dimensional equality selecting on same (record and tag and) occurrence Mathematically, these operators create an intersection of the left hand with a projection of the right hand to the first n dimensions cross product with the full space of the remaining dimensions. E.g. A;B := A intersect {(r,t,x,y)|exist o,p with (r,t,o,p) in B}, which we can denote as the (G) set of B. The full 4-dimensional equality operator, pointer identity, is a pseudo operator as described above. The difference operator - ^ ("NOT" or "WITHOUT") selects based on 1-dimensional inequality, i.e. those pointer from the left hand set for which there exists NO pointer with the same record id in the right hand set. A^B := A intersect {(r,x,y,z)|not exist t,o,p with (r,t,o,p) in B} (the NOT set of B). Note that the literal words OR, AND and NOT are not operators, but terms. The distance operators are based on a proximity relation in the 4th dimension, requiring 3-dimensional equality (same record, tag and occurrence): - .[...] or (n) max distance A sequence of n dots or a number in parentheses selects pointers from the left hand, for which there is a pointer in the right hand where the word position p differs by at most n. (0) is pointer identity. A(n)B := A intersect {(r,t,o,x)|exists p with abs(p-x) <=n and (r,t,o,p) in B} (the dist set of B). - $$[$$] exact distance: likewise with exact position difference n. (A single $ could be confused with prefix match, but is the same as a .). Due to the problems of word counting during filtering, the distance operators may silently behave like 3-dimensional equality. * operator precedence Adjacent terms with no operator in between are combined by *. The operator precedence can by overriden by enclosing any expression in parentheses (). The operators in decreasing precedence: - pointer/key identity - - positional distance . and $$ - field level match , and ; ((F) and (G)) - filter / - record level match * and ^ - additive + On the same precedence level, the distance operators .. and $$ associate right ('A.B.C' is 'A.(B.C)'), all others associate left ('A B C' is '(A*B)*C'). The tag filter / has highest precedence to its right hand, since the tag list is not subject to any operator evaluation. * detailed semantics of searching First of all, tag filters are *defined* to be applied fully distributive. For operators of higher precedence, they are logically distributive: '(A (G) B)/101' yields the same as '(A/101) (G) (B/101)', hence their lower precedence. For '(A ^ B)/100', it looks like we should first select all results for A, strip those where B is (anywhere) in the same record, and then strip all in fields other than 100. This, however, would be better written as 'A/100 ^ B': there is no point in delaying a tag filter. The only reasonable use of applying a tag filter to a lower precedence operation (which requires use of parentheses anyway) is to distribute it. So, '(A ^ B)/100' is *defined* as 'A/100 ^ B/100'. Second, we define tag filters to not nest, but override, so that only the innermost applies. E.g. in '(A/101 B C)/102', only B and C have to be found in field 102, while A is checked in 101. Again, this is because there would be no point in intersecting nested tag lists: it's shorter to write what should be in effect in the first place. To summarize, - tag filters are always distributed down to the term level, where they are applied most efficiently, and - there is always at most one tag filter in effect. In the further reasoning, tag filters are not considered. The final result set of a search expression is 1-dimensional (record id only), in other words Malete does not maintain > http://lcweb.loc.gov/z3950/agency/markup/23.html Extended Result Sets , but discards the other dimensions after evaluating a search. If higher dimensional operators (including tag filter) are applied to #n references, the original search has to be recomputed. (Which might not be supported). The reason for this is: - it reduces space requirements between queries - delivering records from search results is easier, especially support for free cursor movement - commutative properties allow for a couple of optimizations Intermediate results during query evaluation may have to contain higher dimensional data, depending on the operations which will be applied to them: - 1 record id only for OR, AND, NOT - 2 plus tag for ; (G) - 3 plus occurrence for , (F) - 4 plus position in field for .. and $$ We say that an expression is evaluated in a n-dimensional context, if n is the highest dimension of an operator to be applied to its result set. (The OR here is considered neutral in that it keeps data in all dimensions, but does not require it). This is an important property, because it allows to ignore what happens to other dimensions. Not all operators are associative or commutative, in general, the order of evaluation and operands matter. For example, should 'A.B.C' select a C adjacent to A or to B or to any of these? By the right associative property, 'A.B.C' is 'A.(B.C)', which by definition first selects those pointers to (occurrences of) B, which are adjacent to C, then pointers to A adjacent to one of those Bs, which is probably what you expected. It is important for 'B.C' to create pointers to B, not to C, so we can not use 'A.(C.B)' nor '(A.B).C', both of which would select As adjacent to C. * properties of operators Yet, there are some useful properties of operators. The multiplicative operators and the difference are based on equality relations, which are reflexive (p=p), symmetric (if p=q then q=p) and transitive (if p=q and q=r, then p=r). The proximity relation is symmetric, but not transitive: If A is adjacent to B, and B adjacent to C, A is usually not adjacent to C (actually it can only be adjacent to another C like in a field "C A B C"). Exact distance is not even reflexive. Therefore: - intersection operators are (left and right) distributive over union: '(A+B) op C' = '(A op C)+(B op C)' and 'A op (B+C)' = '(A op B)+(A op C)' for all but the NOT, which reads 'A^(B+C)' = 'A^B^C'. - multiplicative operators and NOT are (self) associative: '(A op B) op C' = 'A op (B op C)', where op is one of * , ; or ^. This also holds where the operators are not the same, but the first is not NOT and the second has no higher dimension. - multiplicative operators are relative commutative in their dimension: 'A op B' = 'B op A' if evaluated in no higher dimensional context (since the results are based on a n-dimensional equality, they are equal in the first n dimensions). - for all intersection operators o,p we also have a property similar to associativity: '(A o B) p C' = '(A p C) o B', since both expressions intersect A with both the o-set of B and the p-set of C. In a commutative context, this also equals 'B o (A p C)', which can be used to get an existing left hand result set B inside a right hand subexpression. - distance is commutative in lower dimensional contexts: e.g. '(A.B),C' = '(B.A),C', since it requires 3-dimensional equality, and, with the above, also '(A.B).C' = 'B.(A.C)'. Note that the context dimension depends not only on the direct parent operator, but the highest dimension of any ancestor in the parse tree: While '(A.B),C' = '(B.A),C' as final expressions have the same (1-dimensional) results, '((A.B),C)..D' != '((B.A),C)..D', since D's distance will be tested against A or B, resp. * processing search expressions Some notes on how a search expression is evaluated might be helpful in order to write efficient queries. Basic naive query processing proceeds by - recursively calculating the result sets of subexpressions - combining these result sets according to the operator Techniques used in combining result sets are - merging This is used by the OR to create the union of sets. Duplicates with regard to the required dimensions are eliminated. - marking First calculate the left hand result set. Iterate over the right hand set, look up and mark elements having the relation in the left hand set. Finally, eliminate all unmarked (or marked, for NOT) elements. This can be used to implement all relation based operators. However, it is most useful where the right hand expression can loop its elements without actually creating the set (direct marking). - filtering First calculate the right hand result set. Then, while generating elements for the left hand, look up the relation in the right hand set and store left hand elements only as appropriate. Note that a filter applies only while creating sets, not during marking or merging. All techniques require the result sets to be ordered according to the dimensions, i.e. by record id then tag then occ and pos, to run efficiently. The actual implementation tries to avoid the naive approach. Instead of calculating subexpression results independently, the results of previous calculations and the mode in which they finally will be combined are used where possible. - tag filters are always applied at the lowest (term) level, thus do not require an additional filtering step - the dimensional context is used to purge unnecessary duplicates and, optionally, can be used to reorder expressions according to their properties. Creating result sets: - for a ref, in an one-dimensional search expression, we have the results set. Else, it can be created by processing the query independently or inlining its text. Anyway, in most cases, especially where we create under a filter, we need to create a copy. - for an exact match term, the result set can be directly fetched from the index. Other term relations are an OR over all keys they match. - the result set of an OR is created by creating both parts recursively and merging them. Any filter is applied to all subexpressions. - for all other expressions, if not filtering and the right hand does not support direct marking, create it and use as left hand filter. Else create the left hand (using any filter), evaluate the right hand using direct marking, if possible, else create it and mark. Direct marking in subexpressions: - all simple expressions can efficiently loop their elements and thus support direct marking. - OR and NOT support direct marking, if the left hand supports it and the right hand is a simple expression Notes - OR supports marking also if the right hand is a plain OR. However, this case (A+(B+C)) should be expressed as ((A+B)+C). - marking can be extended to AND and OR, NOT with complex subexpressions using multiple mark values. (G) and (F) can support marking like AND when evaluated in their dimension; the only non-trivial case of this is when embedded in an OR. This extended marking is an optional feature. - in terms of buffers, filtering is most efficient in an intersection, because the filter set can be released early. It is most efficient, where the right hand support direct marking. In associative situations, it may be possible to rewrite '(A op B) op C' as 'A op (B op C)', so C's result can be released early during the evaluation of B and likewise B's result in A. Reordering expressions is by far the most important technique, and the best thing about it is that you can do it. To be most efficient, the parse tree must be as unbalanced as possible: In the best case, it is fully left associative like '((((A op B) op C) op D) op E) op F' (with A-F simple expressions supporting direct marking). That way we can always directly operate at the outermost level on the result set of the previous expression. On the default optimization level 0, no automatic expressions rewriting is done. Higher optimization levels will be specified, e.g. we would expect level 1 to recognize a common case 'A.B.C' in a lower dimensional context, and rewrite the proper expression 'A.(B.C)' to '(B.C).A', so we can reuse B's details once using marking. Full optimization should also use all commutattive properties to choose subexpressions for marking and filtering. * processing filter expressions For filter expressions, the result sets are created from the current record and thus are assumed to be rather small, so that buffer space is not an issue. Instead, short cut evaluation can be assumed to be much more important, since for an interesting filter expression we expect it to not match in most cases. All multiplicative operators should evaluate a term before a complex subexpression and stop processing if it evaluates to the empty set. Probably the most important optimization here would be to detect terms with exact, prefix or contains relation which are required by multiplicative operators and to do a quick check for their occurence as soon as possible. Especially where such a filter is applied to all records of a db, it should be applied while streaming the record file, hopefully approaching the performance of grep. * limits Several limits apply to query and especially search evaluation. Some are currently hardcoded, but may become configuration options in future releases. - the number of subexpressions (terminals and operators) in any expression is limited to 500. - the nesting depth during parsing is limited to 50. (This is usually lower then the depth of the final parse tree). - the number of open queries is limited by session option q (default 20). - the size (number of records) of a search result set is limited by session option s (default 10.000). --- $Id: Query.txt,v 1.19 2004/07/16 12:42:39 istr Exp $ For all those mathematical terms, > http://mathforum.org/dr.math/faq/faq.property.glossary.html ask Dr. Math doc/RecStruct.txt0100644000076400001440000003275610101173706013141 0ustar gabiusersOpenIsis/Malete field definition and record structures * overview A Malete record is a sequence of one or more fields. The first one is called the header, all others are identified by a numeric tag. As far as the Malete database core is concerned, a field may contain any arbitrary bytes but newline characters. Assuming anything about the structure of field data, including any encoding of binary data, is solely at the application's discretion. As Malete is designed to be a multi-purpose database engine, there is no special schema enforced. However, there is a schema suggested and used by the OpenIsis application. In the database's > MetaData metadata record, fields with tag (00)6 are reserved for this purpose (abuse at your own risk). The rationale of this field definition is to provide enough flexibility to efficiently support representations of all structures found in Z39.2 based systems (including but transcending the traditional CDS/ISIS software), especially the various MARC formats, as well as full representations of data commonly stored and transmitted in a couple of other formats like MIME and XML. The term "representation" means that Malete will not bother to directly support XML's angle brackets nor XML's/MIME's foo="bar" options nor the subfield delimiter characters of MARC or CDS/ISIS. Rather, for any such data there should be a lossless transformation to an efficient representation in some format described by this field definition. * structure of fields While fields may be used to hold a single value, it is a common technique to treat them as a sequence of subfields. ("A data element considered as a component of a field.", Z39.2). A field may contain, in that order: - 0 or more positional subfields of fixed length - 1 or more positional subfields of variable length - 0 or more identified subfields of variable length Fixed length subfields end after as many bytes (not characters!) as given by their length. They are typically used for data coded in some ASCII values. Neither UTF-8 characters nor the delimiter character should be stored in fixed length fields (however, it's up to the application to exercise care). Variable length subfields end at a delimiter character or end of field. Malete by default uses a tabulator as delimiter, and import of CDS/ISIS databases converts the caret (hat '^') to tabs, however applications are free to use any delimiter they want. Positional subfields are identified by their position within the field, i.e. by counting that many bytes and delimiters. Of course, there is only one nth position within a field, i.e. every positional subfield can occur at most once. Since the first n bytes and first m delimited subfields are used as the positional subfields, they may be omitted only if end of field is seen, i.e. all other subfields are omitted. Identified subfields, on the other hand, start with a single character identifying the subfield, just like fields in a record are identified by a tag. Applications unaware of UTF-8 may demand a single byte as identifier. Where portability is an issue, only ASCII letters and digits should be used. Since there is at least one positional variable subfield, identified subfields always start after a delimiter (in accordance with Z39.2). An identified subfield may occur zero, one or more times in a field. The MAIN VALUE of a field contains the fixed length subfields together with the first positional variable subfield. Sloppy applications may use anything up to the first delimiter, assuming that fixed subfields do not contain it. In the common situation of having no fixed length subfields, the main value equals the first positional field. The main value in a field is very similar to a record's header and commonly used as a key to select a field in a record. The properties of subfields stated so far are consequences of their very definition. Additional properties, e.g. the main value being empty or an identified subfield having a fixed length a/o occuring exactly once, may be demanded by field definition. It is the applications responsibility to make sure records do not violate the field definition; the Malete server will happily store whatever it receives. * definition of fields The field definition uses fields of the metadata record, one per each field and one per subfield. These fields themselves do not use fixed length subfields. The main value is a (non-unique) key: - 'tag' for a field definition, where tag is an integer. Negative numbers are reserved for counted structures. By convention, general application data fields should > TagUse use tags 100 - 999. - 'tag#len' for a fixed subfield, where len is a positive integer - 'tag#' for an additional variable positional subfield. the first variable positional subfield's type, values and xref are defined with the main field definition. - 'tag^i' for a subfield identified by character i ('^' is the actual hat character, which is NOT the subfield delimiter; the field definition uses tabulators) All other subfields in the field definition are identified and optional: - n name A name by which a field or subfield can be referred to. Field names must be unique and subfield names must be unique in their field. It is strongly recommended to only use C identifiers, i.e. ASCII letters, digits and the underscore, not starting with a digit. - d description Some textual description suitable for the database users. - m min/mandatory The sub/field must occur at least as many times as given by this option's value (empty=1, absent=0). - r repeatable The sub/field must occur at most as many times as given by this option's value (empty=any, absent=1). A value preceeded by '+' (including a single '+' for any) implies the mandatory option (at least one occurrence). - v value Every occurrence of this repeatable option is of the form name=value, associating the symbolic name with a legal value for the sub/field. The first such value is used as a default where the sub/field is created for some reason. - t type Type of this sub/field; see further below. Defaults to any (non-control) characters. Applications might support repeated alternative types. * types of subfields Note that a field's type actually defines the type of its first positional variable subfield (which is usually the main value). If there are no subfields defined for a field, the field's value equals its main value. A simple type definition consists of a single letter indicating a character type, optionally followed by some digits giving a repeat count. Unlike the byte-based length restrictions of fixed length fields, the repeat count should be assumed in terms of characters. For the terms "alphabetic" and "digit", it's up to the application's UNICODE support to properly check these attributes for non-ASCII characters. Simple environments may assume any code greater than 127 alphabetic. Basic character types are: - c character Any character with a code value greater or equal 32 (i.e. no C0 controls). - a alpha Any alphabetic character. - d digit ASCII digits '0'-'9'. - n numeric Digits and optional leading minus sign. - w word Alpha, digits and underscore. Extended character/byte types, possibly not supported by all environments, are: - b bit/boolean ASCII digits '0' or '1'. If a subfield of this type is absent, '0' should be assumed, but a '1' if it's present and empty. - r raw Raw bytes using newline/vertical tab encoding as suggested by the > Protocol - i integer Binary coded fix point decimal numbers using two decimal digits per byte (128-99 .. 128+99) and starting with a byte 144 plus the bytes before the decimal point (minus for negative numbers). Such integers sort properly, avoid newlines and tabs, and the first byte (for up to 30 decimal digits) is not valid in UTF-8 or any ISO charset. - t time Date and time as GTF integer. Up to 8 digits before the decimal point for date YYYYMMDD, after the decimal point hhmmss... For all simple type definitions, the same letter may be used uppercase. With lowercase, the repeat count gives a maximum and defaults to any. With an uppercase type letter, the repeat count is exact and defaults to 1. Complex type definitions include the following: - = pattern Pattern is a sequence of simple type definitions of basic character types. E.g. 'A3a6' denotes 3 to 9 alphabetic characters. Any special character in pattern denotes itself (typically as separator). - ~ regexp Depending on the regexp package used. - " literal Must have one of the values listed with the field's v option. The field definition of basic field definition is: $ 6 6 nfdt dfield definition r t=Nc 6 6^n nname dsub/field name 6 6^d ndesc dsub/field description 6 6^m nmin dmin number of occurrences tn 6 6^r nrep dmax number of occurrences 6 6^v nval dnamed values r 6 6^t ntype dsub/field type $ * advanced field definition There are some advanced field definition options which are probably not supported by all applications. Where used, however, the following formats are recommended: - b base The key or name of another sub/field definition in this metadata record from which options (and, for a field, subfield definitions) should be used for this entity. Obviously just a convenience feature. - x xref Definition of some other entity referred to by the value of this sub/field. Described elsewhere. - s structure a.k.a. subrecord The field introduces a structure in the record; see further below. - c child This repeatable option specifies a tag or name of a legal child field. Applications might support this being followed by '[:min][-max]' to specify a min a/o max count of occurences of this child, or one of the letters '+' (at least once), '?' (at most once), '!' (exactly once) or '*' (any number of times, default). In the definition of those childs, r0 may be used to indicate that they should not occur in the record but where explicitly listed as legal child. * structures The structure option indicates that a field is the header of a structure, indicating that some fields following it in the record somehow belong to it. ("A group of fields within a record that may be treated as a logical entity. (When a record describes more than one entity, the descriptions of individual entities may be treated as subrecords.)", Z39.2). While in general there are a couple of ways to mark a sequence of fields as logically being one entity, there are three methods supported by the field definition: - counted structures If the s option's value is empty, the field's tag is the negative number of fields belonging to the structure, including the header. This is the means used by the > Protocol to efficiently and transparently embed any records in messages. Obviously counted structures cannot be accessed by their tag. They are defined as some negative tags. Some known format of their main value (especially a literal) may be used to access them by key. - delimited structures If the s option's value is '+', the field has one additional initial subfield of fixed length 1. For a given occurence of this field, this subfield must contain either '-', indicating that there are no childs, be absent (i.e. the field is completely empty), or contain a '+', indicating that everything up to a matching empty field of same tag are the structures childs. - fixed structures If the s option's value is a number, the structure has exactly as many childs as given by this number. Note that the number of fields may be greater if the childs are structures themselves. Rarely used. Note that while the field definition in general does not specify the ordering of fields, the childs of a structure are always a consecutive range according to the structure's definition. Z39.2 reserves control field 002 for "subrecord purposes", e.g. listing the offsets of such "groups of fields". * recommendations - fixed subfields should contain only bytes 32 to 126, inclusive - if delimited structures are used, they should be used consistently, i.e. all fields (but 0) should have that type - fixed structures should only be used for internal purposes * examples The headers of email or other MIME messages like $ Subject: hi there Content-Type: text/plain; charset="iso8859-1" $ using a field definition of $ 6 10 nsubject 6 11 ncontent-type 6 11^c ncharset $ map to $ 10 hi there 11 text/plain ciso8859-1 $ Value options could be used to encode common value like text/plain. Using delimited structures, a typical HTML table definition starting with $ ... $ using $ 6 100 ntd s+ 6 100^w nwidth ... 6 101 ntr ... $ will be compacted to $ 100 + w100% p0 s0 m0 h0 t0 l0 b0 101 + 102 + vtop w160 0 this is the textbody 103 - 0 of the td node 102 101 ... $ which could save half of the internet's bandwidth. Some strict XML parsers limit a node to at most one textnode child, which then should be stored in the node's main value. * conformance Most features of Z39.2 (a.k.a. ISO2709 a.k.a. IIF) map directly to Malete records. Subfield identifiers in Z39.2 can use more than one character, however, MARC always uses one. Initial fixed subfields are dubbed "indicators" by Z39.2, MARC uses two of length 1. They are not considered "data elements", as other subfields are. Here, fixed subfields are considered less special. MIME and *ML (SGML,HTML,XML...) data structures can be converted to records in a straightforward manner after a parser has resolved entities and the like. --- $Id: RecStruct.txt,v 1.6 2004/07/26 12:23:34 kripke Exp $ doc/Selene.txt0100644000076400001440000000341710237127061012431 0ustar gabiusersAnnouncing Selene, the successor to Malete and third generation of OpenIsis databases. * History - initial versions of OpenIsis have been using traditional CDS/ISIS file formats (both "DOS" and "UNIX" versions). On the masterfile (.MST) both reading and writing has been implemented, but supporting updates to the index turned out to be too tedious and was not considered worthwhile, given it's inherent limitations. However, one could collect data using OpenIsis and use winisis or the bireme tools to recreate the index. A writable alternate index format (.OXI) and also a text based masterfile format (.TXT) have been developed. - OpenIsis 1.0 ("Malete") was based on these new formats, but using a completely new code base, and supporting import and export to both CDS/ISIS (including the index) and "ISO" (2709, MARC) files. Malete also added support for arbitrarily defined collations even on UTF-8. Like the traditional formats, Malete uses a fixed record model of a sequence of pairs of a number ("tag") and a text value. * Proposed changes in Selene - Selene is a database core library fully independent of CDS/ISIS, supporting arbitrary data (e.g. XML text) and not requiring any particular record model. - Support for binary data requires minor changes to the file formats and protocol allowing explicit data lengths. - The Selene server incorporates the > http://lua.org/ Lua programming language, which can be used e.g. for server based index definitions similar to the traditional field selection table (.FST). - OpenIsis 2 will be using Selene as a basis, adding special support for the CDS/ISIS record model and import/export from/to traditional file formats. Given some funding, we shall also make a new attempt to support the traditional formatting language. doc/Status.txt0100644000076400001440000000352010230226367012476 0ustar gabiusersCurrent status of the Malete engine. The 1.0 release is available for > DownLoad This is probably also the last release of the malete project as such, as basically all goals have been achieved. The follow-up is the Selene project, featuring a wider range of supported data structures and builtin > http://lua.org/ Lua scripting. * what 1.0 contains - malete server communicating over TCP or UNIX sockets as described in the > Protocol - multi process server option (on systems with working fork and fcntl, i.e. not on windows) - search expressions and a single '?' to retrieve record contents as described in the > Query query spec - sample client interface in PHP and possibly other languages - conversion tools > CmdLine from and to traditional CDS/ISIS, IIF (ISO 2709) and plaintext - stable small file formats as > FileFormats specified (32 bit file access using 8 byte pointers) - collation support is working as specified in > CharSet character set support * what 1.0 does not contain (at least not officially) - protocol features marked as optional extensions - record filtering since that is rarely used and can be done at application level - large files and key compression - any promise on the C API (it IS going to change) - the Malete object model definitively needs some more tweaking before being ready for prime time * what is still TO DO - de/serializing of records to/from files accessed in a stream like fashion needs to be debugged, unified and heavily tested. The concept of a sink probably needs to be straightened out. - the single process server should do proper session switching. Every session should have a separate input buffer. On the other hand, we may also dump single process completely. - both single process and multi process server need heavy testing --- $Id: Status.txt,v 1.4 2005/02/23 17:08:00 kripke Exp $ doc/TagUse.txt0100644000076400001440000000514410101202422012366 0ustar gabiusershow to assign tag numbers * standards > IIF (ISO 2709) makes a distinction between - control fields 001-009 which must be the leading fields in a record in order of increasing tags. Control fields do not use any subfields, i.e. no indicators nor subfield delimiters or identifiers. IIF demands exactly one field 001, which must hold a record control number. 002 is reserved for subrecord purposes, others are implementation defined. (IIF also supports alpha tags 00a-00z, which are not supported by ISIS and to my knowledge not used in bibliographical standards). - data fields with arbitrary tags to be in any ordering, but after all control fields. All data fields start with a fixed number of indicators (two in MARC) followed by subfields, each introduced by a delimiter and identifier. Unimarc, Marc21 and even MAB agree on the use of 005 as version identifier, holding a last modified time as YYYYMMDDHHMMSS.T (with tenths of a second) according to ISO 8601. Marc21 additionally uses 003 as "control number identifier" (identifying the organization who assigned 001), 006 for "additional material characteristics", 007 for physical description and 008 for "general information" like date and place of publication. Other 0xx tags are typically used for various other identifiers and classifications of the record like an ISBN, a number according to some national bibliography, dewey decimal classification and so on. * recommendations It is suggested to use (00)1 and (00)5 as record "control number" (unique primary key) and last modified time. Probably it is also not such a bad idea of IIF to reserve (00)2 for "subrecord purposes". If you are storing bibliographical or similar data, or the number of fields is in the hundreds for some other reason, it is suggested to assign tags in nice blocks of 100 like > http://www.ifla.org/VI/3/p1996-1/sec-uni.htm Unimarc does, using 0xx for identifiers. Typically you will want to stick to IIF's restrictions on control fields vs. data fields only if you are actually using some MARC format. Yet, some future special support for a primary key in field 001 might require it to be the first field. In other typical database applications like storing addressbook data, email headers or shop orders, use (0)10 - (0)99, possibly arranged in blocks of 10 (e.g. personal, postal, phone and so on). Use 3,4 and 6-9 for what you somehow consider "control fields" or to save some digits if you really are going to get by with 6 "data fields". Use 0 for unqualified text. Do not use negative tags other than for counted subrecords. --- $Id: TagUse.txt,v 1.2 2004/07/26 13:20:18 kripke Exp $ doc/Tcl.txt0100644000076400001440000000555110124533547011746 0ustar gabiuserssketch of the malete Tcl binding * records are proper Tcl data objects A record is represented as a standard Tcl data object. It can be assigned to a variable, a slot in an array or to a proc's return value (the interp's result). The objects's string rep is the standard serialization (without trailing blank line; blank lines are ignored when de-serializing). The object's internal rep is a Malete List, i.e. a modifiable record. * records are accessed by views Just like Tcl image objects are accessed using the command "image", there are several commands to access records. Different commands interpret the records in different ways: A basic command provides a plain list view on the array, while derived extensions may provide a view on > RecStruct delimited structures or other specialized record types. Unlike the "image" command, record views are stateful. They may - be bound to a "current" record object - maintain a cursor position in the current record - have a current frame (subrecord) - have an associated field definition to resolve symbolic names - have an associated database name (defaulting to the view's name) The view commands may be cloned in order to maintain several such states in parallel. A different view command may be used for every record resulting in a Tk-style "object oriented" feel. The typical application, however, will use only a small number of views like "v" and "w" or some database names. The actual work of accessing field definitions and databases will be handed over to additional configurable procedures, which are typically implemented in Tcl. * view command Many commands take an index expression , which - selects some of the record's fields - cuts some part out of these fields, typically subfields When invoked with no parameters, the current record object is returned. Most commonly used in rendering record contents is the one-parameter form v _index_, returning a field based on the cursor and current record. In all other forms, the first parameter denotes a view function: - list _index_ returns all data addressed by _index_ as a list - array _index_ returns all data addressed by _index_ as a names and values list like the Tcl command "array get" - head _index_ get parts of header - with _record_ [args ...] Without args, binds the view to _record_. With args, temporarily binds the view to _record_ for the execution of the view command given by args. - do _script_ execute script. Syntactic sugar for with _record_ do _script_ Modifying records: In the basic implementation, a view may only modify a record if it's the only view bound to this record. As the record's struct Tcl_Obj provides a twoPtrValue from which only one pointer is needed to refer to the list, the ptr2 may be used to point to a view for synchronization purposes (and views bound to the same record may be chained). Database access: read, write (ins/upd) mlt/Makefile0100644000076400001440000000716410233731425012150 0ustar gabiusers# #/* # The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. # Version 0.9.x (patchlevel see file Version) # Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org # # This library is free software; you can redistribute it and/or # modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 2.1 of the License, or (at your option) any later version. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the GNU Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # see README for more information #EOH */ # # $Id: Makefile,v 1.2 2005/04/27 15:43:17 kripke Exp $ # the Makefile # ifndef WIN32 O=../bin EXE= RUN= else O=../win EXE=.exe RUN=wine -- include ../Mingw.mk endif MALETE=$(O)/malete$(EXE) # standard make defines ifndef CC CC=gcc endif # DBG= ifndef NDEBUG DBG=-g STRIP= else DBG=-DNDEBUG STRIP=-s endif # ARCH=-march=i586 # compile CPPFLAGS=-I. # note that -O is necessary for -Wuninitialized warnings # -fstack-check ??? CFLAGS=$(DBG) -O2 -Wall -W -Wshadow -Winline # ############################# link ################################## SYSLIBS= ifeq (solaris,$(OSTYPE)) # rt for nanosleep SYSLIBS=-lrt -lsocket -lnsl endif ifdef WIN32 # SYSLIBS=-lws2_32 SYSLIBS=-lwsock32 endif # ############################# targets ################################## # # standard default target all: malete .PHONY: all clean malete testdb # $Id: Makefile,v 1.2 2005/04/27 15:43:17 kripke Exp $ # core features ifdef WIN32 NOSHMODE=1 endif ifndef NOSHMODE DEF_SHMODE=-DBUILD_SHMODE endif ifdef LARGEFILES DEF_LARGE=-DBUILD_LARGE endif # tool features ifdef NOCDS CDS= DEF_CDS=-DBUILD_NOCDS else CDS=cds ifdef CDSUNIX DEF_CDS=-DBUILD_CDSUNIX endif endif ifdef NOIIF IIF= DEF_IIF=-DBUILD_NOIIF else IIF=iif endif DEFS=$(DEF_SHMODE) $(DEF_LARGE) $(DEF_CDS) $(DEF_IIF) OBJS=uti sys rdx qdx cdx db pw dbo hash srv $(CDS) $(IIF) $(O)/pw.o: pw.h $(O)/dbo.o: pw.h $(O)/hash.o: pw.h $(O)/srv.o: pw.h tool.h $(O)/cds.o: pw.h tool.h $(O)/iif.o: pw.h tool.h $(O)/%.o: %.c core.h @- [ -d $(O) ] || mkdir $(O) $(CC) -c $(CPPFLAGS) $(DEFS) $(CFLAGS) -o $@ $< clean: rm -rf ../bin ../win ../test/test.* [ ! -f core ] || rm core $(MALETE): $(O)/malete.o $(OBJS:%=$(O)/%.o) $(CC) $(DBG) $(STRIP) -o $@ $^ $(SYSLIBS) # ########################## demos and tests ############################### # malete: $(MALETE) $(RUN) $(MALETE) ../test/test.m0d: ../test/cds.m0d : "copying cds options ..." cp $< $@ ../test/test.mrd: ../test/cds.mrd : "copying cds records ..." cp $< $@ ../test/test.mqd: ../test/cds.mqt : "loading cds index ..." $(RUN) $(MALETE) qload -vi test -f../test/ -wa <../test/cds.mqt ../test/test.mrx: ../test/test.mrd : "updating access file/s" [ ! -f $@ ] || rm $@ $(RUN) $(MALETE) dbinfo test -f../test/ -wa ../test/test.mqx: ../test/test.mqd ../test/test.m0d : "updating access file/s" [ ! -f $@ ] || rm $@ $(RUN) $(MALETE) dbinfo test -f../test/ -wa cleandb: rm -rf ../test/test.* testdb: ../test/test.m0d ../test/test.mrd ../test/test.mqd \ ../test/test.mrx ../test/test.mqx @ls -l $^ server: $(MALETE) testdb $(RUN) $(MALETE) server -vd -S test -f../test/ # ############################# rules ################################## # # $@ target $^ all prerequisites $? newer $< first $* stem # mlt/cds.c0100644000076400001440000005644410175233513011432 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: cds.c,v 1.1 2005/01/24 17:58:35 kripke Exp $ malete CDS/ISIS conversion */ #include "tool.h" /* structs with 4-byte aligned ("UNIX") or packed (Intel "DOS"+WinIsis) ints */ #ifdef CPU_BIG_ENDIAN # define BUILD_CDSUNIX /* no packed version on SPARCs, M68K or PPCs */ #endif #ifdef BUILD_CDSUNIX # define PACKED #else # ifdef __GNUC__ # define PACKED __attribute__((packed)) # else # error "please define how to pack" # endif #endif enum { CDS_MST, CDS_XRF, /* master file, cross ref */ CDS_CNT, CDS_IFP, /* index control and record pointers */ /* do not change this order: see f2 */ CDS_N01, CDS_L01, /* 10 byte nodes and leaves */ CDS_N02, CDS_L02, /* 30 byte nodes and leaves */ CDS_FILES }; static const char *cds_ext[] = { "mst", "xrf", "cnt", "ifp", "n01", "l01", "n02", "l02" }; typedef file cds_files[CDS_FILES]; typedef struct { /* master file control header */ int ctlm; /* should be 0 */ int nmfn; /* next avail MFN */ int nmfb; /* next free block, 1st = 1 */ short nmfp; /* next free pos (in block) */ short type; /* 0 (1 is for "messages" mst) */ int rcnt; /* record count; nmfn-1 - #deleted */ int mfx1; /* unused */ int mfx2; /* unused; "LAN lock" */ int mfx3; /* unused; "LAN lock" */ } Mfc; typedef struct { /* field dictionary in master file record */ unsigned short tag; unsigned short pos; unsigned short len; } Dct; typedef struct { /* master file record */ int mfn; short mfrl; int bwb PACKED; short bwp; short base; short nvf; short stat; Dct dict[1]; } Mfr; /* it is common belief, that the first 4 fields (12 bytes packed, 14 aligned) are to be in one 512-byte block */ enum { MFR_BASEOFF = ((char*)&((Mfr*)0)->base) - (char*)0, MFR_DICTOFF = ((char*)&((Mfr*)0)->dict) - (char*)0 }; typedef struct { /* cross ref block */ int xpos; int xrec[127]; } Xrf; typedef struct { /* index control */ short type; /* 1 or 2 of N0x */ short ordn; /* 5 */ short ordf; /* 5 */ short n; /* 15 */ short k; /* 5 */ short lev; /* depth: N0x levels under root (root only: 0) */ int posr; /* root pos */ int nmax; /* N0x blocks */ int fmax; /* L0x blocks */ short abno; /* 0 (!) if "abnormal" (root only) */ } Cnt; #ifdef BUILD_CDSUNIX /* while Cnt layout is independent of alignment ... */ # define CNT_SIZE 28 /* ... sizeof(Cnt) is not */ #else # define CNT_SIZE 26 #endif typedef struct { /* "posting" pointer to record */ char b[8]; /* 3+2+1+2 byte rid+tag+occ+pos big endian */ } Pst; typedef struct { /* inverted file postings header */ int nxtb; /* continuation */ int nxtp; int totp; /* total postings */ int segp; /* actually in this segment <= totp */ int segc; /* segment capacity >= segc */ /* Pst pst[...]; possibly spanning several Ifb blocks */ } Ifp; typedef struct { /* inverted file block */ int ifpblk; int ifprec[127]; /* ... */ } Ifb; #define NSTRUCT(n) struct { \ int pos; \ short ock; \ short it; \ struct { \ char key[n]; \ int ref PACKED; \ } idx[10]; \ } typedef NSTRUCT(10) N01; typedef NSTRUCT(30) N02; #define LSTRUCT(n) struct { \ int pos; \ short ock; \ short it; \ int ps; \ struct { \ char key[n]; \ int infb PACKED; \ int infp PACKED; \ } idx[10]; \ } typedef LSTRUCT(10) L01; typedef LSTRUCT(30) L02; static const unsigned char cp850[] = { /* Latin1 values of Cp850 upper half */ 199,252,233,226,228,224,229,231,234,235,232,239,238,236,196,197, 201,230,198,244,246,242,251,249,255,214,220,248,163,216,215,'?', 225,237,243,250,241,209,170,186,191,174,172,189,188,161,171,187, '?','?','?','?','?',193,194,192,169,'?','?','?','?',162,165,'?', '?','?','?','?','?','?',227,195,'?','?','?','?','?','?','?',164, 240,208,202,203,200,'?',205,206,207,'?','?','?','?',166,204,'?', 211,223,212,210,245,213,181,254,222,218,219,217,253,221,175,180, 173,177,'?',190,182,167,247,184,176,168,183,185,179,178,'?',160 }; typedef struct { Fld base; /* points into env->opt */ int notab; /* do not convert ^ <-> tabs */ int novt; /* do not convert newline <-> vtabs */ int no850; /* do not convert Cp850 <-> ISO Latin1 */ int write; /* write cds */ } CdsOpt; static int cdsOpt (CdsOpt *opt, Fld *args, Db *db) { Fld o; memset(opt, 0, sizeof(*opt)); for ( o.val = 0; vGet(&o, args, "BN"); ) switch (o.tag) { case 'B': opt->base = o; continue; case 'N': if (VEQZ(&o, "otab")) opt->notab = 1; else if (VEQZ(&o, "ovt")) opt->novt = 1; else if (VEQZ(&o, "o850")) opt->no850 = 1; else return eRr(ERR_INVAL, "bad cds opt '-N%.*s'", o.len, o.val); } if ( !opt->base.len ) opt->base.len = strlen(opt->base.val = db->pat); return 0; } /* cdsOpt */ int cdsuse () { eRr(ERR_INVAL, "[cdsimp|cdsexp]:\n" "-Bbase: path/basename of cds db (default = malete db)\n" "-Notab: do not convert ^ <-> tab\n" "-Novt: do not convert newline <-> vtab\n" "-No850: do not convert Cp850 <-> ISO Latin1" ); return ERR_INVAL; } /* cdsuse */ static int cdsOpen (cds_files cds, CdsOpt *opt) { int filmode = opt->write ? FIL_RDWR|FIL_CREAT|FIL_TRUNC : FIL_RD; int l = opt->base.len, i; char *path = mAlloc(l+5); memcpy(path, opt->base.val, l); path[l++] = '.'; for (i=0; i= mfc.nmfn ) t = mfc.nmfn-1; for (mfn=f; mfn<=t; mfn++) { int pos = xrf.xrec[mfn-f]; int mfb = (pos >> 11); union { Mfr mfr; char buf[0x7fff]; } da; Dct *d = da.mfr.dict; char *b, *e; if ( 0 > mfb ) { eRr(LOG_TRACE, "rec %d was deleted", mfn); continue; } /* ignore bits 9, 10 signaling pending index update */ pos &= 0x1ff; pos |= (mfb-1)<<9; got = fPread(cds+CDS_MST, da.buf, sizeof(Mfr), pos); if ( got < MFR_DICTOFF ) CLEANUP((ERR_TRASH, "rec %d too short %d", mfn, got)); if ( da.mfr.mfrl < 0 ) { eRr(LOG_WARN, "rec %d pos %d locked", mfn, pos); da.mfr.mfrl = -da.mfr.mfrl; } if ( da.mfr.mfrl < (int)(MFR_DICTOFF+da.mfr.nvf*sizeof(Dct)) ) CLEANUP((ERR_TRASH, "rec %d pos %d bad mfrl %d nvf %d", mfn, pos, da.mfr.mfrl, da.mfr.nvf)); got = fPread(cds+CDS_MST, da.buf, da.mfr.mfrl, pos); if ( da.mfr.mfrl != got ) { if ( got == -da.mfr.mfrl ) da.mfr.mfrl = -da.mfr.mfrl; else CLEANUP((ERR_TRASH, "rec %d pos %d got %d bytes want %d", mfn, pos, got, da.mfr.mfrl)); } b = da.buf+da.mfr.base; e = da.buf+da.mfr.mfrl; lReset(&rec); for ( ; da.mfr.nvf--; d++ ) { char *p = b+d->pos, *q = p+d->len; if ( p < b || q > e ) CLEANUP((ERR_TRASH, "bad pos %d len %d field %d", d->pos, d->len, d-da.mfr.dict)); for ( ; ptag, b+d->pos, d->len); } dWrite(db, rec.fld, mfn); } } /* mst done */ /* we assume first leaf has pos 1; maybe we should use N0x to search */ l01.ps = l02.ps = 1; l01.ock = l02.ock = n1 = n2 = 0; ifbs[0].ifpblk = ifbs[1].ifpblk = 0; for (;;) { /* loop leaves */ Fld use; int infb, infp, got, more = 0; Ifp ifp; Ifb *ifb; if ( n1 == l01.ock && (l01.pos = l01.ps) ) { if ((int)sizeof(l01) != (got = fPread(cds+CDS_L01,&l01,sizeof(l01),(l01.ps-1)*sizeof(l01))) ) CLEANUP((ERR_TRASH, "l01 %d got %d", l01.ps, got)); eRr(LOG_TRACE, "l01 pos %d ock %d nxt %d", l01.pos, l01.ock, l01.ps); n1 = 0; if ( !l01.ock ) continue; } if ( n2 == l02.ock && (l02.pos = l02.ps) ) { if ((int)sizeof(l02) != (got = fPread(cds+CDS_L02,&l02,sizeof(l02),(l02.ps-1)*sizeof(l02))) ) CLEANUP((ERR_TRASH, "l02 %d got %d", l02.ps, got)); eRr(LOG_TRACE, "l02 pos %d ock %d nxt %d", l02.pos, l02.ock, l02.ps); n2 = 0; if ( !l02.ock ) continue; } if ( !l01.pos && !l02.pos ) break; if ( n1 == l01.ock ) use.tag = 1; else if ( n2 == l02.ock ) use.tag = 0; else if ( 0 > memcmp(l02.idx[n2].key, l01.idx[n1].key, 10) ) use.tag = 1; else use.tag = 0; if ( use.tag ) { infb = l02.idx[n2].infb; infp = l02.idx[n2].infp; use.val = l02.idx[n2++].key; use.len = 30; } else { infb = l01.idx[n1].infb; infp = l01.idx[n1].infp; use.val = l01.idx[n1++].key; use.len = 10; } ifb = ifbs+use.tag; /* kill trailing blanks */ while ( use.len && ' ' == use.val[use.len-1] ) use.len--; eRr(LOG_TRACE, "'%.*s'", use.len, use.val); memcpy(key.byt, use.val, key.len = use.len); key.val.len = db->qdx.vsz; while (infb) { Pst *pst; int n; if ( ifb->ifpblk != infb && (512 != fPread(cds+CDS_IFP,ifb,sizeof(Ifb),(infb-1)*512) || ifb->ifpblk != infb) ) CLEANUP((ERR_TRASH, "ifp %d",infb)); if ( more ) { pst = (Pst*)ifb->ifprec; if ( (n = 127/2) > more ) n = more; more -= n; eRr(LOG_TRACE, "+%d @%d", n, infb); } else { if ( infp > 127-5 ) CLEANUP((ERR_TRASH, "bad infp %d for infb %d", infp, infb)); ifp = *(Ifp*)(ifb->ifprec + infp); pst = (Pst*)(ifb->ifprec + infp + 5); if ( (n = (127-5-infp)/2) > ifp.segp ) n = ifp.segp; more = ifp.segp - n; eRr(LOG_TRACE, "ifp @%d,%d: %d %d %d %d %d", infb, infp, ifp.nxtb, ifp.nxtp, ifp.totp, ifp.segp, ifp.segc); } if ( more ) infb++; else { infb = ifp.nxtb; infp = ifp.nxtp; } for ( ; n--; pst++ ) { if ( QDX_ISIS == db->qdx.ptr ) memcpy(key.val.byt, pst->b, 8); else { Ptr ptr; ptr.ext = 0; ptr.rid = pst->b[0]<<16 | pst->b[1]<<8 | pst->b[2]; ptr.tag = pst->b[3]<<8 | pst->b[4]; ptr.pos = pst->b[5]<<16 | pst->b[6]<<8 | pst->b[7]; qMkVal(&key.val, &ptr, db->qdx.ptr); } qLoad(&db->qdx, &key); } } /* while infb */ } key.val.len = 0; qLoad(&db->qdx, &key); cleanup: #undef CLEANUP cdsClose(cds); return ret; } /* cdsimp */ /* export index */ typedef struct { QLoop qlp; file *cds; int totp; /* in current segment */ int infb; /* of current ifp */ int infp; /* of current ifp */ Ifb ifb; /* current block; flushed on demand */ int rec; /* next in ifb.ifprec 0..126 */ L01 l01; /* new keys are immediatly inserted here ... */ L02 l02; /* ... structs flushed on demand */ N01 n01; N02 n02; } CdsLoop; static void closeifp (CdsLoop *self) /* write Ifp header */ { Ifp buf, *ifp; if ( self->infb != self->ifb.ifpblk ) /* Ifp not in current blk */ ifp = &buf; else ifp = (Ifp*)(self->ifb.ifprec+self->infp); ifp->nxtb = ifp->nxtp = 0; /* used only for partial updates */ ifp->totp = ifp->segp = ifp->segc = self->totp; /* diff only after split */ if ( self->infb != self->ifb.ifpblk ) fPwrite(self->cds+CDS_IFP, ifp, sizeof(Ifp), ((self->infb-1)<<9) + 4*(self->infp+1)); self->totp = 0; /* hmmmm ... From the manual: Cuando se carga secuencialmente el archivo invertido (por ejemplo despues de una generacion completa del archivo invertido con ISISINV), cada lista esta formada por uno o mas segmentos adyacentes. Si IFPTOT <= 32768, entonces: IFPNXTB/IFPNXTP = 0/0 y IFPTOT = IFPSEGP = IFPSEGC. This seems to imply that numbers really should be used as signed shorts!? */ } /* closeifp */ static void closeifb (CdsLoop *self) { eRr(LOG_TRACE, "w ifb %d", self->ifb.ifpblk); fPwrite(self->cds+CDS_IFP,&self->ifb,sizeof(Ifb),(self->ifb.ifpblk-1)<<9); self->ifb.ifpblk++; self->rec = 0; } /* closeifb */ static int cdsCb (CdsLoop *self) { Pst *pst; unsigned i, vsz = self->qlp.qdx->vsz; const unsigned char *v = self->qlp.vals; if ( !(QSAME & self->qlp.flg) ) { /* add new key -- new segment */ L01 *l0x; /* might really be a L02 */ int l, n, oref; /* key length, offset of ref int(s) in idx */ int f2, sl, *ref; char *key; if ( self->totp ) closeifp(self); if ( self->rec > 127-5 ) closeifb(self); if ( (l = self->qlp.cur.len) <= 10 ) { /* l01 entry */ l0x = &self->l01; f2 = 0; n = 10; sl = sizeof(L01); } else { l0x = (L01*)&self->l02; f2 = 2; if ( l > (n = 30) ) l = 30; sl = sizeof(L02); } #ifdef BUILD_CDSUNIX oref = n+2; #else oref = n; #endif if ( 10 == l0x->ock ) { /* need new block */ l0x->ps = l0x->pos+1; eRr(LOG_TRACE, "w l0 n %d pos %d", n, l0x->pos); fPwrite(self->cds+CDS_L01+f2, l0x, sl, (l0x->pos-1)*sl); l0x->pos++; l0x->ock = 0; } key = l0x->idx[0].key + l0x->ock*(oref+8); /* sizeof idx */ memset(key, ' ', n); memcpy(key, self->qlp.cur.byt, l); ref = (int*)(key+oref); ref[0] = self->infb = self->ifb.ifpblk; ref[1] = self->infp = self->rec; self->rec += 5; /* reserve space for Ifp header */ if ( !l0x->ock++ ) { /* also write node entry */ N01 *n0x = 10==n ? &self->n01 : (N01*)&self->n02; int sn = 10==n ? sizeof(N01) : sizeof(N02); char *nkey; if ( 10 == n0x->ock ) { eRr(LOG_TRACE, "w n0 n %d pos %d", n, n0x->pos); fPwrite(self->cds+CDS_N01+f2, n0x, sn, (n0x->pos-1)*sn); n0x->pos++; n0x->ock = 0; } nkey = n0x->idx[0].key + n0x->ock++*(oref+4); if ( 2 == n0x->pos && 1 == n0x->ock ) memset(nkey, ' ', n); /* 1st key - start of level */ else memcpy(nkey, key, n); *(int*)(nkey+oref) = -l0x->pos; } } pst = (Pst*)(self->ifb.ifprec + self->rec); eRr(LOG_TRACE, "+ %d ptr @ %d.%d", self->qlp.nvals, self->ifb.ifpblk, self->rec); for ( i=self->qlp.nvals; i--; self->rec+=2, pst++, v += vsz) { if ( self->rec > 125 ) { closeifb(self); pst = (Pst*)self->ifb.ifprec; } if ( QDX_ISIS == self->qlp.qdx->ptr ) memcpy(pst->b, v, 8); else { Ptr ptr; qRdVal(&ptr, v, self->qlp.qdx->ptr); pst->b[0] = ptr.rid >> 16; pst->b[1] = ptr.rid >> 8; pst->b[2] = ptr.rid; pst->b[3] = ptr.tag >> 8; pst->b[4] = ptr.tag; pst->b[5] = ptr.pos >> 16; pst->b[6] = ptr.pos >> 8; pst->b[7] = ptr.pos; } } self->totp += self->qlp.nvals; return 0; } /* cdsCb */ int cdsexp (Db *db, Fld *args) { CdsOpt opt; cds_files cds; int ret = 0, pos, i, j; Mfc mfc; List rec; char recode[256]; Xrf xrf; CdsLoop clp; Cnt cnt[2]; if ( cdsOpt(&opt, args, db) ) return cdsuse(); opt.write = 1; if ( cdsOpen(cds, &opt) ) return eRr(ERR_INVAL, "error creating CDS base"); eRr(LOG_INFO, "exporting %d recs", db->rdx.mid); lInit(&rec, ""); /* prepare recode table */ for (i=0; i<128; i++) recode[i] = (char)i; if ( !opt.notab ) recode[TAB] = '^'; if ( !opt.novt ) recode[VT] = LF; if ( opt.no850 ) for (i=128; i<256; i++) recode[i] = (char)i; else { for (i=128; i<256; i++) recode[i] = '?'; for (i=160; i<256; i++) /* reverse cp850 */ for (j=0; j<128; j++) if ( i == cp850[j] ) { recode[i] = j+128; break; } } xrf.xpos = 1; mfc.rcnt = 0; /* record count; nmfn-1 - #deleted */ pos = 64; /* as of folklore, 1st rec should start here */ for (i=1; i<=db->rdx.mid; i++) { /* loop records */ int xref; if ( 0 >= dRead(lClr(&rec), db, i) ) { /* no rec */ eRr(LOG_TRACE, "rec %d was deleted", i); xref = -1 << 11; } else { union { Mfr mfr; char buf[0x7fff]; } da; char *b, *p, *e = da.buf+sizeof(da.buf); Dct *d = da.mfr.dict, *de; Fld *f, *fe = LEND(&rec); unsigned avl = 0x7fff - MFR_DICTOFF; xref = pos & 511; /* pos in block */ /* the manual explicitly demands 0-498 and forbids 500 */ if ( xref > 498 /*512 - MFR_BASEOFF = 500 packed */ ) { pos += 512; /* advance to next block */ pos &= ~511; xref = 0; } xref |= ((pos >> 9)+1) << 11; da.mfr.mfn = i; da.mfr.bwb = 0; da.mfr.bwp = 0; da.mfr.base = 0; da.mfr.nvf = 0; da.mfr.stat = 0; for ( f=rec.fld; ++f= f->len ) avl -= f->len; else avl = 0; } p = b = da.buf + (da.mfr.base = MFR_DICTOFF + da.mfr.nvf*sizeof(Dct)); de = d + da.mfr.nvf; for ( f=rec.fld; dval, *ve; d->tag = f->tag; d->pos = p-b; d->len = e-p < (int)f->len ? e-p : (int)f->len; for ( ve = v+d->len; v < ve; ) *p++ = recode[*v++]; } da.mfr.mfrl = p - da.buf; eRr(LOG_VERBOSE, "write rec %d len %d pos %d", i, da.mfr.mfrl, pos); fPwrite(cds+CDS_MST, da.buf, da.mfr.mfrl, pos); if ( 1 & (pos += da.mfr.mfrl) ) /* round to even */ pos++; mfc.rcnt++; } xrf.xrec[j = (i-1)%127] = xref; if ( i == db->rdx.mid ) { /* last */ while ( ++j < 127 ) xrf.xrec[j] = 0; xrf.xpos = -xrf.xpos; } else if ( 126 != j ) continue; eRr(LOG_VERBOSE, "%d: write Xrf %d", i, xrf.xpos); fWrite(cds+CDS_XRF, &xrf, sizeof(Xrf)); xrf.xpos++; } mfc.ctlm = 0; /* should be 0 */ mfc.nmfn = i; /* next avail MFN */ mfc.nmfb = pos >> 9; /* next free block, 1st = 1 */ mfc.nmfp = pos & 511; /* next free pos (in block) */ mfc.type = 0; /* 0 (1 is for "messages" mst) */ mfc.mfx1 = 0; /* unused */ mfc.mfx2 = 0; /* unused; "LAN lock" */ mfc.mfx3 = 0; /* unused; "LAN lock" */ fPwrite(cds+CDS_MST, &mfc, sizeof(mfc), 0); /* mst done */ memset(&clp, 0, sizeof(clp)); clp.qlp.qcb = (QCb*)cdsCb; clp.cds = cds; clp.ifb.ifpblk = 1; clp.l01.pos = clp.l02.pos = 1; clp.n01.pos = clp.n02.pos = 2; /* start at 2; 1 will hold the root */ clp.l01.it = clp.n01.it = 1; clp.l02.it = clp.n02.it = 2; clp.qlp.qdx = &db->qdx; qLoop(&clp.qlp); eRr(LOG_VERBOSE, "loop got pos ifp %d l01 %d l02 %d n01 %d n02 %d", clp.ifb.ifpblk, clp.l01.pos, clp.l02.pos, clp.n01.pos, clp.n02.pos); if ( clp.totp ) closeifp(&clp); if ( clp.rec ) closeifb(&clp); clp.l01.ps = clp.l02.ps = 0; /* no next */ fPwrite(cds+CDS_L01, &clp.l01, sizeof(L01), (clp.l01.pos-1)*sizeof(L01)); fPwrite(cds+CDS_L02, &clp.l02, sizeof(L02), (clp.l02.pos-1)*sizeof(L02)); for ( i=2; i--; ) { cnt[i].type = i+1; cnt[i].ordn = 5; cnt[i].ordf = 5; cnt[i].n = 15; cnt[i].k = 5; cnt[i].lev = 0; cnt[i].posr = 1; cnt[i].nmax = 1; cnt[i].abno = 1; } cnt[0].fmax = clp.l01.pos; cnt[1].fmax = clp.l02.pos; if ( 2 == clp.n01.pos ) { /* the poor one and only node */ /* since we did not open a 2nd node, this one was not written */ clp.n01.pos = 1; /* make it the root */ cnt[0].abno = 0; /* really the field means "normal", not "abnormal" */ } fPwrite(cds+CDS_N01, &clp.n01, sizeof(N01), (clp.n01.pos-1)*sizeof(N01)); if ( 2 == clp.n02.pos ) { clp.n02.pos = 1; cnt[1].abno = 0; } fPwrite(cds+CDS_N02, &clp.n02, sizeof(N02), (clp.n02.pos-1)*sizeof(N02)); /* build the trees */ if ( cnt[0].abno ) { int f = 2, t = clp.n01.pos; /* from - to on level */ for (;;) { N01 c, p; /* child, parent */ int start = t+1; cnt[0].lev++; p.pos = 10 >= (t-f) ? 1 : start; p.ock = 1; /* blank key ref to f */ p.it = 1; memset(p.idx[0].key, ' ', sizeof(p.idx[0].key)); p.idx[0].ref = f; for ( i=f+1; i<=t; i++ ) { fPread(cds+CDS_N01, &c, sizeof(c), (i-1)*sizeof(c)); eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - %d:'%.*s'->%d", cnt[0].lev-1, c.pos, sizeof(c.idx[0].key), c.idx[0].key, c.idx[0].ref, c.ock, sizeof(c.idx[0].key), c.idx[c.ock-1].key, c.idx[c.ock-1].ref); if ( 10 == p.ock ) { eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - '%.*s'->%d", cnt[0].lev, p.pos, sizeof(p.idx[0].key), p.idx[0].key, p.idx[0].ref, sizeof(p.idx[0].key), p.idx[9].key, p.idx[9].ref); fPwrite(cds+CDS_N01, &p, sizeof(p), (p.pos-1)*sizeof(p)); p.pos++; p.ock = 0; } memcpy(p.idx[p.ock].key, c.idx[0].key, sizeof(c.idx[0].key)); p.idx[p.ock++].ref = i; } eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - %d:'%.*s'->%d", cnt[0].lev, p.pos, sizeof(p.idx[0].key), p.idx[0].key, p.idx[0].ref, p.ock, sizeof(p.idx[0].key), p.idx[p.ock-1].key, p.idx[p.ock-1].ref); fPwrite(cds+CDS_N01, &p, sizeof(p), (p.pos-1)*sizeof(p)); if ( 1 == p.pos ) break; f = start; cnt[0].nmax = t = p.pos; } } if ( cnt[1].abno ) { int f = 2, t = clp.n02.pos; /* from - to on level */ for (;;) { N02 c, p; /* child, parent */ int start = t+1; cnt[1].lev++; p.pos = 10 >= (t-f) ? 1 : start; p.ock = 1; /* blank key ref to f */ p.it = 2; memset(p.idx[0].key, ' ', sizeof(p.idx[0].key)); p.idx[0].ref = f; for ( i=f+1; i<=t; i++ ) { fPread(cds+CDS_N02, &c, sizeof(c), (i-1)*sizeof(c)); eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - %d:'%.*s'->%d", cnt[1].lev-1, c.pos, sizeof(c.idx[0].key), c.idx[0].key, c.idx[0].ref, c.ock, sizeof(c.idx[0].key), c.idx[c.ock-1].key, c.idx[c.ock-1].ref); if ( 10 == p.ock ) { eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - '%.*s'->%d", cnt[1].lev, p.pos, sizeof(p.idx[0].key), p.idx[0].key, p.idx[0].ref, sizeof(p.idx[0].key), p.idx[9].key, p.idx[9].ref); fPwrite(cds+CDS_N02, &p, sizeof(p), (p.pos-1)*sizeof(p)); p.pos++; p.ock = 0; } memcpy(p.idx[p.ock].key, c.idx[0].key, sizeof(c.idx[0].key)); p.idx[p.ock++].ref = i; } eRr(LOG_TRACE, "lev %d blk %d '%.*s'->%d - %d:'%.*s'->%d", cnt[0].lev, p.pos, sizeof(p.idx[0].key), p.idx[0].key, p.idx[0].ref, p.ock, sizeof(p.idx[0].key), p.idx[p.ock-1].key, p.idx[p.ock-1].ref); fPwrite(cds+CDS_N02, &p, sizeof(p), (p.pos-1)*sizeof(p)); if ( 1 == p.pos ) break; f = start; cnt[1].nmax = t = p.pos; } } fWrite(cds+CDS_CNT,cnt,CNT_SIZE); fWrite(cds+CDS_CNT,cnt+1,CNT_SIZE); cdsClose(cds); return ret; } /* cdsexp */ mlt/cdx.c0100644000076400001440000005036710175233531011435 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: cdx.c,v 1.1 2005/01/24 17:58:49 kripke Exp $ charset collation */ #include "core.h" /* We use - p codes 0,1,..p-1 for primaries (word and nonword) - v codes p..p+v-1 for secondary and tertiary level variants - m codes p+v..p+v+m-1 for maps (targets) We index code numbers to an array[p+v+m] of unsigned ints as code values. The highest bit is the nonword indicator. The lower nibble of the highest byte is the length of the code's byte sequence (for maps in #code values, else in # cleartext bytes). If the sequence does not fit into the lower 3 bytes, the value&0xffffff is the offset of the bytes. For primaries and variants, the bytes are cleartext. For maps, the bytes are code text (length given as #codes), where every code uses one byte, if p+v<=256, else two. With variants, we also have an array[p+v] of variant info. For a primary code 0..p-1, this is the number of secondary and tertiary variants, and the code of the first of the (sec+1)*(ter+1) variants - 1. For a variant code p..p+v-1, this is it's secondary and tertiary weights and the code of the associated primary. */ typedef union { unsigned u; struct { #ifdef CPU_BIG_ENDIAN unsigned char hi; unsigned char c[3]; #else unsigned char c[3]; unsigned char hi; #endif } b; } Cv; typedef struct Var { /* variant info */ unsigned char sec; /* secondary variant */ unsigned char ter; /* tertiary variant */ unsigned short rel; /* code of related variant */ } Var; /* For the encoding, we map every byte to: - it's code number - the "word" bit - the index of a table of possible following bytes To cover the BMP with UTF-8, we need 1072 tables: - 32 bytes 110* initiating a 2 byte sequence, using one table each - 16 bytes 1110* initiating a 3 byte sequence, each having a table with 64 second bytes 10*, using 16+16*64 = 1040 tables (- the 32 tables used for the UTF-16 surrogates D800-DFFF) CJK 3400-9FFF alone uses 432 tables (some 27.000 ideographs) */ typedef struct Bin { /* byte info */ unsigned short cod; /* byte's collation code */ unsigned short tab; /* table */ } Bin; typedef Bin Bins [256]; /* full table */ typedef struct Tab { /* table info */ unsigned off; /* bytes offset of Bin for min */ unsigned char min; unsigned char max; unsigned short unu; } Tab; enum { BIT_VARIANTS = 0x20, BIT_FRENCH = 0x10, TYP_PLAIN = 0, TYP_VARIANTS = BIT_VARIANTS, /* TODO */ TYP_FRENCH = BIT_VARIANTS|BIT_FRENCH, /* TODO */ CVLAT1 = 104 /* # primary codes for builtin lat1cdx */ }; /* header of a dumped/mapped cx */ struct Cdx { unsigned char mag[3]; /* magic MCX or mcx for mapped cdx */ unsigned char typ; /* base type | bits per (primary) code-1 */ /* currently only 8(7) and 16(15) bits supported */ unsigned short pri; /* p # primary codes incl. 0,1 */ unsigned short var; /* p + v # variants */ unsigned short map; /* p+v + m # maps */ unsigned short tab; /* # non-root tables 1..tab */ /* redundant offsets */ unsigned ovi; /* offset variant_infos */ unsigned otp; /* offset table_pointers - 4 */ unsigned siz; /* total size */ Bins bt0; /* root table, always full */ unsigned cv[CVLAT1]; /* actually Cv code_values[map=p+v+m] */ /* Var variant_infos[var=p+v], if BIT_VARIANTS */ /* Tab table_pointers[tab] */ /* Bin byte_tables ... */ /* unsigned char *more_bytes */ }; static const char MAGIC[3] = #ifdef CPU_BIG_ENDIAN {'M','C','X'}; # define W(b) (0x01000000|(b)<<16) # define N(b) (0x81000000|(b)<<16) #else {'m','c','x'}; # define W(b) (0x01000000|(b)) # define N(b) (0x81000000|(b)) #endif #if 0 const Cdx lat1cdx = { {'s','t','a'}, /* mark as static */ TYP_PLAIN|7, CVLAT1, CVLAT1, CVLAT1, 0,0,0,0, { /* byte infos */ #define B(b) {b,0} /* 32 C0 controls */ B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), /* ! " # $ % & ' */ B( 2),B( 3),B( 4),B( 5),B( 6),B( 7),B( 8),B( 9), /* ( ) * + , - . / */ B(10),B(11),B(12),B(13),B(14),B(15),B(16),B(17), /* 0 1 2 3 4 5 6 7 */ B(18),B(19),B(20),B(21),B(22),B(23),B(24),B(25), /* 8 9 : ; < = > ? */ B(26),B(27),B(28),B(29),B(30),B(31),B(32),B(33), /* @ A B C D E F G */ B(34),B(35),B(36),B(37),B(38),B(39),B(40),B(41), B(42),B(43),B(44),B(45),B(46),B(47),B(48),B(49), /* H-O */ B(50),B(51),B(52),B(53),B(54),B(55),B(56),B(57), /* P-W */ /* X Y Z [ \ ] ^ _ */ B(58),B(59),B(60),B(61),B(62),B(63),B(64),B(65), /* ` a b c d e f g */ B(66),B(35),B(36),B(37),B(38),B(39),B(40),B(41), B(42),B(43),B(44),B(45),B(46),B(47),B(48),B(49), /* h-o */ B(50),B(51),B(52),B(53),B(54),B(55),B(56),B(57), /* p-w */ /* x y z { | } ~ DEL */ B(58),B(59),B(60),B(67),B(68),B(69),B(70),B(0), /* 32 C1 controls */ B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), B(0),B(0),B(0),B(0),B(0),B(0),B(0),B(0), /* 32 mostly symbols */ B( 2),B(71),B(72),B(73),B(74),B(75),B(76),B(77), B(78),B(79),B(80),B(81),B(82),B(83),B(84),B(85), B(86),B(87),B(88),B(89),B(90),B(91),B(92),B(93), B(94),B(95),B(96),B(97),B(98),B(99),B(100),B(101), /* 64 Latin alphas including 2 symbols */ B(35),B(35),B(35),B(35),B(35),B(35),B(35),B(37), /* 7A,1C */ B(39),B(39),B(39),B(39),B(43),B(43),B(43),B(43), /* 4E,4I */ B(54),B(49),B(49),B(49),B(49),B(49),B(49),B(102), /* TN5O* */ B(49),B(55),B(55),B(55),B(55),B(59),B(54),B(53), /* O4UYTS */ B(35),B(35),B(35),B(35),B(35),B(35),B(35),B(37), /* 7a,1c */ B(39),B(39),B(39),B(39),B(43),B(43),B(43),B(43), /* 4e,4i */ B(54),B(49),B(49),B(49),B(49),B(49),B(49),B(103), /* tn5o% */ B(49),B(55),B(55),B(55),B(55),B(59),B(54),B(59) /* o4uyty */ #undef B }, { /* code values, all using immediate bytes */ 0, N(9), N(32),N(33),N(34),N(35),N(36),N(37),N(38),N(39), /* !"#$%&' */ N(40),N(41),N(42),N(43),N(44),N(45),N(46),N(47), /* ()*+,-./ */ W(48),W(49),W(50),W(51),W(52),W(53),W(54),W(55), /* 01234567 */ W(56),W(57),N(58),N(59),N(60),N(61),N(62),N(63), /* 89:;<=>? */ N(64),W(65),W(66),W(67),W(68),W(69),W(70),W(71), /* @ABCDEFG */ W(72),W(73),W(74),W(75),W(76),W(77),W(78),W(79), /* HIJKLMNO */ W(80),W(81),W(82),W(83),W(84),W(85),W(86),W(87), /* PQRSTUWW */ W(88),W(89),W(90),N(91),N(92),N(93),N(94),N(95), /* XYZ[\]^_ */ N(96), /*a-z*/ N(123),N(124),N(125),N(126), /* ` {|}~ */ N(161),N(162),N(163),N(164),N(165),N(166),N(167), N(168),N(169),N(170),N(171),N(172),N(173),N(174),N(175), N(176),N(177),N(178),N(179),N(180),N(181),N(182),N(183), N(184),N(185),N(186),N(187),N(188),N(189),N(190),N(191), N(215), N(247) } }; /* lat1cdx */ #endif /* encode l byte in b to key. key->len is the max len on input, resulting len on return. stop if max key len is hit or on word boundary, if words. return #used bytes in b. */ int cEnc ( const Cdx *cdx, Key *key, unsigned char *b, int l, int words ) { const unsigned char * const base = (unsigned char *)cdx; const Bin * const bt0 = cdx->bt0; const Tab * const tp = (Tab*)(base + cdx->otp); const int bits = 1+(0xf&cdx->typ), mapcodeshift = 256var ? 1 : 0; const unsigned char *e = b+l; int bitsleft = 8*key->len, pfxlen = 0; int unassigned = 0; Bin seq[CDX_MAXSEQ], *top; /* current sequence */ unsigned u, len; unsigned short code; Cv cv; key->len = 0; for ( ; b < e && bits <= bitsleft; b++ ) { eRr(LOG_TRACE, "byte %c of %d bits %d/%d", *b, e-b, bitsleft, bits); /* traverse byte info tables to find longest matching sequence */ for (*(top = seq) = bt0[*b]; top->tab && b < e; ) { const Tab * const t = tp + top->tab; unsigned char n = b[1]; LOG_DBG(LOG_DEBUG, "table %d check %d %d-%d off %d %d", top->tab, n, t->min, t->max, t->off, *(int*)(base+t->off)); if ( n < t->min || n > t->max /* out of bounds */ || !(u = ((unsigned*)(base+t->off))[n - t->min]) /* unassigned */ ) break; *++top = *(Bin*)&u; b++; } for ( ; top > seq && !top->cod; top-- ) /* no complete sequence */ b--; code = top->cod; eRr(LOG_TRACE, "code %d", code); if (words) { if ( 1= words /* ok, passed it for the 1st time */ ) { pfxlen = key->len; words = 1; } else if (0x80000000&cdx->cv[code]) { /* hit nonword */ if (pfxlen == key->len) /* skip to word */ continue; break; /* had something after prefix */ } } if (cdx->pri > code) { if ( code ) unassigned = 0; else { /* unassigned */ if ( unassigned ) continue; unassigned = 1; code = 1; } switch (bits) { case 16: key->byt[key->len++] = code >> 8; /* always bigend */ case 8: key->byt[key->len++] = (char)code; } bitsleft -= bits; continue; } /* if (cdx->var > top->cod) variant */ /* else map: */ cv.u = cdx->cv[code]; if ((len = (0xf & cv.b.hi)< (bitsleft -= (len << 3)) ) break; memcpy(key->byt+key->len, ~3&len ? base+(0xffffff&cv.u) : cv.b.c, len); key->len += len; } } if (/*words &&*/ key->len == pfxlen) /* if pfxlen: found no words */ key->len = 0; return b+l-e; } /* cEnc */ int cDec ( const Cdx *cdx, unsigned char *b, int l, Key *key ) { const unsigned char * const base = (unsigned char *)cdx; const unsigned char *k = key->byt, *e = b+l-CDX_MAXSEQ; const int bits = 1+(0xf&cdx->typ); int bitsleft = 8*key->len; Cv cv; for ( ; b < e && bits <= bitsleft; bitsleft -= bits ) { unsigned short v = *k++; if ( 16 == bits ) v = v<<8 | *k++; cv.u = cdx->cv[v]; switch ( cv.b.hi &= 0xf ) { /* discard word bit */ case 1: *b++ = cv.b.c[0]; continue; case 2: *b++ = cv.b.c[0]; *b++ = cv.b.c[1]; continue; case 3: *b++ = cv.b.c[0]; *b++ = cv.b.c[1]; *b++ = cv.b.c[2]; continue; } memcpy(b, base+(cv.u&0xffffff), cv.b.hi); b += cv.b.hi; } return b+l-CDX_MAXSEQ-e; } /* cDec */ typedef struct { Cdx cdx; unsigned cvb[0x10000-CVLAT1]; Var *vi, vib[0x10000]; Tab tp[1100]; Bins *bt, btb[1100]; unsigned char *p, b[200000]; } CdxMake; /* make byte table entries */ static int mapSeq (CdxMake *mk, unsigned char *p, int len, unsigned short code, int save) { Bins *bt = &mk->cdx.bt0; LOG_DBG(LOG_DEBUG, "map '%.*s' -> %d", len, p, code); if (save) { Cv *cv = (Cv*)mk->cdx.cv+code; if (15 < len) { eRr(ERR_INVAL, "sequence '%.*s' too long %d", len, p, len); return 1; } cv->b.hi = (0x80&save) | len; if (!(~3 & len)) memcpy(cv->b.c, p, len); else { cv->u |= mk->p - mk->b; memcpy(mk->p, p, len); mk->p += len; } } for (;; p++) { Bin *b = *bt+*p; if (!--len) { /* last byte -- assign code */ if (b->cod) { eRr(ERR_INVAL, "attempt to reassign code %d for %d", code, b->cod); return 1; } b->cod = code; return 0; } if (b->tab) bt = mk->btb + (b->tab - 1); else { bt = mk->bt++; b->tab = mk->bt - mk->btb; LOG_DBG(LOG_DEBUG, "new table %d", b->tab); } } return 0; } /** compile collation src */ static Cdx *cMake ( const Fld *src ) { const Fld *eof = REND(src), *f; CdxMake mk; int skipalias = 1, mapcodeshift; unsigned short firstcode = 1; /* of last row */ unsigned bins=0, off, u; Cdx *cdx; LOG_DBG(LOG_DEBUG, "cMake %d", sizeof mk); memset(&mk, 0, sizeof mk); mk.vi = mk.vib; mk.bt = mk.btb; mk.p = mk.b; mk.cdx.cv[0] = 0x80000000; /* 0: unassigned */ mk.cdx.cv[1] = 0x81000009; /* 1: tab */ mk.cdx.pri = 2; /* first pass: gather primaries and aliases */ for (f = src; ++ftag) { unsigned char *p = (unsigned char*)f->val, *e, *t, save; unsigned short code; if (3 > f->len || TAB != p[1]) continue; e = p + f->len; switch (*p) { case 'A': if (skipalias) continue; save = 0; goto alias; case 'W': save = 1; break; case 'N': save = 0x80; break; case 'S': case 'T': skipalias = 1; default: continue; } firstcode = mk.cdx.pri; skipalias = 0; alias: code = firstcode; for (p+=2; e > p; p = t+1) { if (!(t = memchr(p, TAB, e-p))) t = e; if (t > p && mapSeq(&mk, p, t-p, code, save)) return 0; if (save || code < mk.cdx.pri-1) code++; } if (save) mk.cdx.pri = code; } mk.cdx.typ = TYP_PLAIN|(256 < mk.cdx.pri ? 15 : 7); mk.cdx.var = mk.cdx.pri; /* second pass: gather variants */ for (f = src; ++ftag) { unsigned char *p = (unsigned char*)f->val; if (3 > f->len || TAB != p[1]) continue; switch (*p) { case 'S': case 'T': eRr(ERR_INVAL, "this version does not support multilevel"); return 0; } } if (mk.cdx.var > mk.cdx.pri) mk.cdx.typ |= BIT_VARIANTS; mk.cdx.map = mk.cdx.var; /* third pass: resolve maps */ mapcodeshift = 256tag) { unsigned short code, codes[CDX_MAXSEQ]; unsigned char ccodes[CDX_MAXSEQ], *pcodes; unsigned char *p = (unsigned char*)f->val, *e, *t; unsigned short n = 0; Bin seq[CDX_MAXSEQ], *bin; /* current sequence */ Cv *cv; if (3 > f->len || 'M' != p[0] || TAB != p[1]) continue; e = p + f->len; p += 2; if (!(t = memchr(p, TAB, e-p))) continue; for ( ; ptab && p < t;) { /* slightly simpler than in cEnc, since we have full tables */ int *ent = (int*)mk.btb[bin->tab-1]; if (!ent[p[1]]) break; /* unassigned */ *++bin = *(Bin*)(ent + *++p); if (seq+CDX_MAXSEQ == bin) break; } for (; bin>seq && (!bin->cod || mk.cdx.var<=bin->cod); bin--) p--; if (mk.cdx.var>bin->cod) /* ignore map codes */ codes[n++] = bin->cod ? bin->cod : 1; } p = t+1; /* got n codes ... */ if (1 == n) /* single code ~ alias */ code = codes[0]; else { /* make entry cv[mk.cdx.map++] (even for n=0) */ if (mapcodeshift) { pcodes = (unsigned char*)codes; /* native */ #ifndef CPU_BIG_ENDIAN { unsigned short *us = codes+n; while (us-- > codes) *us = (*us>>8) | (*us<<8); /* mk be */ } #endif } else { for (u=n; u--;) ccodes[u] = codes[u]; pcodes = ccodes; } cv = (Cv*)mk.cdx.cv + mk.cdx.map; cv->b.hi = n; if (n && 0x80000000&mk.cdx.cv[codes[0]]) /* inherit from 1st code */ cv->b.hi |= 0x80; n <<= mapcodeshift; if (!(~3 & n)) memcpy(cv->b.c, pcodes, n); else { cv->u |= mk.p - mk.b; memcpy(mk.p, pcodes, n); mk.p += n; } code = mk.cdx.map++; } for (; e > p; p = t+1) { /* map 'em all to mk.cdx.map */ if (!(t = memchr(p, TAB, e-p))) t = e; if (t > p && mapSeq(&mk, p, t-p, code, 0)) return 0; } } /* finish: compact */ mk.cdx.tab = mk.bt - mk.btb; mk.cdx.otp = mk.cdx.ovi = (((char *)&((Cdx*)0)->cv) - (char*)0) /* offsetoff(cv) */ + mk.cdx.map*sizeof (unsigned); if (mk.cdx.var > mk.cdx.pri) mk.cdx.otp += mk.cdx.var * sizeof (Var); off = mk.cdx.otp + mk.cdx.tab * sizeof (Tab); for (u=0; uvar > cdx->pri) memcpy((char*)cdx + cdx->ovi, mk.vib, mk.cdx.var*sizeof (Var)); if (cdx->tab) { memcpy((char*)cdx + cdx->otp, mk.tp, mk.cdx.tab*sizeof (Tab)); for (u=0; uotp -= sizeof (Tab); /* adjust 0 based */ return cdx; } /* cMake */ /* list of open shared cdx */ typedef struct Foo Foo; struct Foo { char nln; /* namelen */ char nam[31]; Foo *foo; int ref; Cdx *cdx; FMap fm; }; #if 0 static Foo lat1foo = { 7, "Latin-1", 0, 1, (Cdx*)&lat1cdx }; static Foo *fools = &lat1foo; #endif static Foo *fools = 0; /** open or compile collation src */ const Cdx *cOpen (const Fld *src) { char *nam = 0, *p; int nln = 0, coldef = 0; Foo *foo = fools; Cdx *cdx = 0; const Fld *eof, *f; char fname[20]; FMap fm; if (src) for (f = src, eof = REND(src); ++f < eof; ) if (MET_COL == f->tag) { coldef = 1; if (2 < f->len && 'C' == f->val[0] && TAB == f->val[1] && TAB != f->val[2] ) { /* named */ nam = f->val+2; nln = f->len-2; if ((p = memchr(nam, TAB, nln))) nln = p-nam; if (nln > 15) nln = 15; LOG_DBG(LOG_DEBUG, "collation name '%.*s'", nln, nam); for (; foo; foo = foo->foo) if (nln == foo->nln && !memcmp(nam, foo->nam, nln)) { /* got it */ foo->ref++; /* ref it */ return foo->cdx; /* ret it */ } /* TODO: try to map -- check type */ memset(fname, 0, sizeof fname); memcpy(fname, nam, nln); memcpy(fname+nln, ".mcx", 5); memset(&fm, 0, sizeof fm); fm.fil = FIL_NONE; if (!fMOpen(&fm, fname, FIL_RD)) { int size = fSize(fm.fil); fm.lim = (size + env.psz-1)>>env.psh; if ( (int)fm.lim != fMap(&fm, fm.lim) || memcmp(MAGIC, fm.map, 3) || size != (int)((Cdx*)fm.map)->siz ) { eRr(ERR_TRASH, "bad coll file '%s'", fname); fMClose(&fm); } else { cdx = (Cdx*)fm.map; eRr(LOG_VERBOSE, "mapped coll '%s' %d bytes", fname, size); #ifndef WIN32 fClose(&fm.fil); /* don't need the file open */ #endif } } } } if (!cdx && (!coldef || !(cdx = cMake(src)))) return 0; if (nam) { /* foo an mmaped or named made */ Foo *newfoo = mAlloc(sizeof *newfoo); memcpy(newfoo->nam, nam, newfoo->nln = nln); newfoo->foo = fools; newfoo->ref = 1; newfoo->cdx = cdx; if (fm.map) /* was mapped */ newfoo->fm = fm; else { /* try to write */ file fil = FIL_NONE; if (!fOpen(&fil, fname, FIL_WR|FIL_CREAT|FIL_TRUNC)) { memcpy(cdx->mag, MAGIC, 3); fWrite(&fil, cdx, cdx->siz); fClose(&fil); eRr(LOG_INFO, "saved coll '%s' %d bytes", fname, cdx->siz); } memcpy(cdx->mag, "nam", 3); /* yet we keep using our selfmade copy */ } fools = newfoo; } return cdx; } /* cOpen */ void cClose (Cdx *cdx) { Foo *foo, *f = 0; switch (cdx->mag[0]) { case 0: /* internal anonymous */ mFree(cdx); case 's': /* internal static */ return; #ifdef CPU_BIG_ENDIAN case 'M': #else case 'm': /* mapped - always named */ #endif case 'n': /* internal named */ for (foo = fools; foo; foo = (f = foo)->foo) if (foo->cdx == cdx) { if (!--foo->ref) { if ('n' == cdx->mag[0]) mFree(cdx); else fMClose(&foo->fm); if (f) f->foo = foo->foo; else fools = foo->foo; mFree(foo); } return; } /* panic time ? */ } } /* cClose */ mlt/core.h0100644000076400001440000010672410207134337011613 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ #ifndef CORE_H #include /* various str* and mem* */ /* $Id: core.h,v 1.2 2005/02/23 17:09:51 kripke Exp $ full interface of the Malete core */ #define CORE_VERSION "1.0" /* ****************************************************************** */ /* */ /* COMPILER FEATURES */ /* */ /* ****************************************************************** */ #ifndef __STDC__ /* defined by ISO C */ #error "need ANSI/ISO C compiler" #endif /* "cpp -dM val, (f)->len ) /* field has primary value key length l */ #define VKEY( f, k, l ) ( \ (f)->len >= (l) && !memcmp( (f)->val, k, l ) \ && ((f)->len == (l) || TAB == (f)->val[(l)]) \ ) /* dup a field as 0-terminated string */ #define VDUPZ( f ) ((char*)mDupz((f)->val, (f)->len)) /* snip tab-separated subfields from value, setting tag to the subfield identifier and len/val to the contents. if opt is not 0, it lists the options wanted (* for the primary). dst->val should be 0-initialized; if it's greater than src->val, search starts at dst->val+dst->len, @return 0 if not found, else 1+dst->len */ extern int vGet ( Fld *dst, const Fld *src, const char *opt ); /* access to the primary value never fails, if initialized, so return len */ #define VPRI( dst, src ) ((dst)->val = 0, vGet(dst, src, 0), (dst)->len) #define VEQZ( f, str ) (!strncmp((f)->val,str,(f)->len) && !str[(f)->len]) /** undo the encoding of lBin (see below). return #bytes int dst, which is <= src->len extern int vDecod ( char *dst, const Fld *src ); */ /* sign(a - b) */ extern int vCmp ( const Fld *a, const Fld *b ); /* a > b */ extern int vGt ( const Fld *a, const Fld *b ); /* comparision function type for rSort */ typedef int VGt ( const Fld *a, const Fld *b ); /* r (record) functions on an array of fields they expect the tag of the first field to be the negative number of fields. The value of the first field ("header") may contain various meta info. For lists representing database "records", the canonical format is [no[@pos]][leader]. Leader can be used e.g. to hold a Z39.2 leader as used by MARC. Other (protocol) lists should start with some type identifier. it's a matter of taste whether you like and use this typedef. as it's not going to be changed but only here to clarify the intend, the core functions do not use it. */ typedef const Fld *Rec; #define RLEN( r ) (-(r)->tag) #define REND( r ) ((r)-(r)->tag) #define RLAST( r ) (REND(r)-1) extern unsigned rSiz ( const Fld *r ); /** get occurence of field. @param pos if given, the first occ starting at pos is searched and pos is set to one after the found position (or after end). may be used to loop all on an int var initialized to 0. @return the field or 0 */ extern const Fld *rGet ( const Fld *r, int tag, int *pos ); /* get field with given tag and primary value or empty pv */ extern const Fld *rKey ( const Fld *r, int tag, const char *key ); /* create a new const rec in a contigous mAlloced piece of mem if siz is 0, rSiz() is used */ extern const Fld *rDup ( const Fld *src, unsigned siz ); /** flatten (serialize) record to tag\tval lines ended by a blank line. buf must be of size rSiz(rec) + 13*RLEN(rec) for sign+10digits+tab+nl + 1 for the blank line @return # of bytes written */ extern int rSer ( char *buf, const Fld *rec ); /** sort fields WARNING: this cuts fields longer than 2GB !!! NOTE that rSort, unlike qsort, does NOT use a 3-way cmp function, but a boolean gt function */ extern void rSort ( Fld *rec, VGt *gt ); extern void rSortTag ( Fld *rec ); extern void rSortVal ( Fld *rec ); /* == rSort(vGt) */ #define DEFBLKLEN 8000 /* default buffer block length */ #define DEFFIELDS 40 /* default number of fields */ typedef struct LBlk { /* chained buffer block */ struct LBlk *nxt; unsigned siz; char byt[DEFBLKLEN]; /* actual len may vary */ } LBlk; /* A full-fledged, modifiable list. */ typedef struct List { Fld *fld; /* fields list, initially fl0 */ unsigned fav; /* fields available at end of *fld buffer */ int siz; /* used secondary buffer size minus holes (add buf-blk.byt) */ char *buf; /* pointing into blk->byt */ char *end; /* of blk->byt */ LBlk *blk; /* buffer chain, initially bl0 */ char *bok; /* if == buf, buffers contain the serialization */ Fld fl0[DEFFIELDS]; LBlk bl0; } List; #define LLEN( l ) RLEN((l)->fld) #define LEND( l ) REND((l)->fld) #define LLAST( l ) RLAST((l)->fld) /* initialize list and set header. if fmt is 0, no header is printed, and the list left empty. (i.e. the first field added will become the header). @return l */ extern List *lInit ( List *l, const char *fmt, ... ); /* completely clear all fields and buffers. */ extern List *lClr ( List *l ); /* clear all fields and buffers, but keep the header. */ extern List *lReset ( List *l ); /* clear all fields and buffers, keep nothing */ extern void OPT_STDCALL lFini ( List *l ); /* increase available space. If buffer has a flusher, call it. fields -1 means we want to append (keep last field contigous, do NOT flush). fields >0 means reserve at least mode fields. */ extern int OPT_REGPARM lExtend ( List *l, unsigned need, int fields ); /* bytes available at buf */ #define LAVL( l ) ((unsigned)((l)->end - (l)->buf)) #define LSIZ( l ) ((unsigned)((l)->buf - (l)->blk->byt + (l)->siz)) /* fragment used to start a new field with tag t. If the list was empty, the field will become the header, ignoring the tag. */ #define LDEFNEWF( l, t ) \ int __i = (l)->fld->tag; \ Fld *__f = (l)->fld - __i--; \ __f->tag = t; \ (l)->fld->tag = __i; /* add field tag t reserving n bytes space */ #define LNEWF( l, t, n ) do { \ if ( ((l)->fav && LAVL(l) >= n) || lExtend( l, n, 1 ) ) { \ LDEFNEWF( l, t ) \ __f->val = (l)->buf; \ __f->len = 0; \ (l)->fav --; \ } } while(0) /* add field tag t using n bytes prefilled space must be preextended for one field */ #define LPREF( l, t, n ) do { \ if ( (l)->fav ) { \ LDEFNEWF( l, t ) \ __f->val = (l)->buf; \ __f->len = n; \ (l)->fav --; \ (l)->buf += n; \ } } while(0) /* add field tag t with value v n bytes long */ #define LADD( l, t, v, n ) do { \ if ( ((l)->fav && LAVL(l) >= n) || lExtend( l, n, 1 ) ) { \ LDEFNEWF( l, t ) \ memcpy( __f->val = (l)->buf, v, __f->len = n ); \ (l)->fav --; \ (l)->buf += n; \ } } while(0) /* append value v n bytes long */ #define LAPP( l, v, n ) do { \ if ( LAVL(l) >= n || lExtend( l, n, -1 ) ) { \ Fld *__f = LLAST(l); \ memcpy( (l)->buf, v, n ); \ __f->len += n; \ (l)->buf += n; \ } } while(0) /* add field tag t with int value i */ #define LADDI( l, t, i ) do { \ if ( ((l)->fav && LAVL(l) >= 12) || lExtend( l, 12, 1 ) ) { \ LDEFNEWF( l, t ) \ (l)->buf += __f->len = i2a( __f->val = (l)->buf, i ); \ (l)->fav --; \ } } while(0) #define LADDS( l, t, s ) LADD( l, t, s, strlen(s) ) #define LAPPS( l, s ) LAPP( l, s, strlen(s) ) #define LADDF( l, f ) LADD( l, f->tag, f->val, f->len ) /** add a field with arbitrary binary data. The encoded field value will not contain newlines. The encoding maps a VT to two bytes VT 0, and a LF to VT 1, if the LF is followed by a 0 or 1, a single VT else. extern void lBin ( List *l, int tag, const char *bytes, int len ); */ /* append canonical options fields. args starting with '-' are treated as options and appended as tab separated subfield w/o the '-'. If the first arg does not start with '-', it is used as primary value. Other non-options are appended as subfields indicated by '@'. In order to skip the program's name, from main() use with argc-1,argv+1. */ extern int lArgv ( List *l, int tag, int argc, const char **argv ); /* same using 0-terminated list. extern int lArgs ( List *l, int tag, const char *arg, ... ); */ /** A variation on this theme as used for env.var: an initial non-option as command and following options go to the header. every following non-option starts a new field. */ extern List *lVar ( List *l, int argc, const char **argv ); /* if siz is 0, rSiz() is used */ extern int lCpy ( List *l, const Fld *src, unsigned siz ); #define LCPY( l, src ) lCpy( l, (src)->fld, LSIZ(src) ) /* create canned version of list as new record */ #define LCAN( l ) rDup( (l)->fld, LSIZ(l) ) /* Print a field to list. a fmt of 0 indicates varargs. */ extern int lOut ( List *to, int tag, const char *fmt, ... ); /* set the first field with tag to val (of len, -1 for strlen) kill additional fields with tag if no field found, add one at end extern int lSet ( List *l, int tag, const char *val, unsigned len ); */ enum { /* list parse state */ LPS_SOR, /* 0, at start of record */ LPS_SOL = 0x1000000, /* at start of line */ LPS_TAG = 0x2000000, /* in tag (only digits seen in line) */ LPS_VAL = 0x3000000, /* in val (some non-digit seen) */ LPS_NEG = 0x4000000, /* found '-' at start of line */ LPS_CR = 0x8000000, /* kill carriage return character */ LPS_LEN = 0x0ffffff }; /** add serialized "text" to rec lps is the buffer length (of up to 16MB) + state bits @return new state, if buffer was exhausted, or #remaining bytes (|LPS_SOR), if a blank line was seen */ extern int lParse (List *l, const char *txt, int lps); /** record sink. The sink function may be called with eor 0 after adding one or several fields to optionally flush partial records. It must be called with eor when a record is complete. In that case it must prepare List to receive a new header. This typically is done by flushing and resetting list, however, it would be sufficient to prepare for a new embedded subrecord by recording the position of the next head. */ typedef struct Sink Sink; typedef void sink (Sink *that, int eor); /* clean all or part of buffer(s) */ struct Sink { List lst; sink *snk; /* sinking function */ void *dst; /* destination */ int off; /* start of current record in list, negative after partial flush */ }; #define SINK(s) (s)->snk(s, 0) #define SEOR(s) (s)->snk(s, 1) #define SCPY(s, l) do { LCPY(&(s)->lst, l); SEOR(s); } while (0) #define SCPYR(s, r) do { lCpy(&(s)->lst, r, 0); SEOR(s); } while (0) /* ****************************************************************** */ /* */ /* ENVIRONMENT AND SYSTEM */ /* */ /* ****************************************************************** */ #ifndef WIN32 typedef int file; #define FIL_NONE -1 #else typedef void *file; #define FIL_NONE 0 #endif /* environment. */ typedef struct Env { List *opt; /* options (header is packed command line) */ file in; Sink *out; /* a proper record sink expecting complete records */ Sink *err; /* a usually field buffered sink for eOut messages */ int log; /* level */ int flg; int wri; /* write mode none/excl/shared */ int sig; /* interrupted by signal */ unsigned psz; /* system page size = 1<lst, t, v, n) #define EAPP(v, n) LAPP(&env.out->lst, v, n) #define EADDS(t, s) LADDS(&env.out->lst, t, s) #define EAPPS(s) LAPPS(&env.out->lst, s) #define EADDI(t, i) LADDI(&env.out->lst, t, i) #define EADDF(f) LADDF(&env.out->lst, f) /* Append a field with abs(tag) to env.out for non-negative tags, else to env.err if tag >= env's loglevel. For system errors, additional info is appended. env.err's eof is called, if any. returns tag supports only a small subset of printf formats -- see the src has %b (bytes), which is like %.*s, but prints the string in hex */ extern int eOut ( int tag, const char *fmt, ... ); extern int eRr ( int tag, const char *fmt, ... ); #ifndef NDEBUG # define LOG_DBG eRr #elif defined( __GNUC__ ) # define LOG_DBG( args... ) #else # define LOG_DBG (void) /* compiler should dispose statement off */ #endif /** Init the env. The first field specifies general command and options. Following fields describe databases. If no sinks are provided, file sinks on stdout and stderr are used. */ extern void cInit ( List *args, Sink *out, Sink *err ); /* typical usage from main() */ #define CINIT(argl) cInit(lVar(lInit(argl,0), argc-1, argv+1), 0, 0) /** memory management. All memory will be initialized, and all allocs but TryAlloc exit the process when out of memory. DO NOT MIX with other alloc/free routines. */ extern void *mAlloc ( int size ); extern void *mDup ( const void *str, int size ); /* cp siz bytes, append a 0 byte */ extern char *mDupz ( const char *str, int size ); extern void *mTryAlloc ( int size ); extern void mFree ( void *mem ); extern Fld *mFldAlloc ( int nfields ); extern void mFldFree ( Fld *fld ); extern LBlk *mBlkAlloc ( int size ); extern void mBlkFree ( LBlk *blk ); extern List *mListAlloc ( const char *name ); /* lInitialized with name */ extern void mListFree ( List *l ); #define mFldAlloc(n) ((Fld*)mAlloc((n)*sizeof(Fld))) #define mFldFree mFree #define mBlkFree mFree #define MFREE(p) do { mFree(p); (p)=0; } while(0) /** set tm to current time, return difference in millis */ extern int tUpd ( lolo *tm ); /** print generalized time yyyyMMddHHmmss + 0 byte to buffer if tm is 0, current time is used if *tm is 0, *tm is updated buffer must have 15 bytes return millis */ extern int tGtf ( char *buf, lolo *tm ); /** grok the fine manual. like tGtf, but with additional 3 digits millis return buffer, which must have 18 bytes */ extern char *tGtfm ( char *buf, lolo *tm ); /** nanosl */ extern void tSleep ( lolo tm ); /* ************************************************************ disk files (block devices) */ enum { /* basic open flags */ FIL_RD = 0x001, /* shall be opened for input */ FIL_WR = 0x002, /* shall be opened for output */ FIL_RDWR = 0x003, /* shall be opened for both */ FIL_TRY = 0x004, /* do not complain if open fails */ /* write flags */ FIL_CREAT = 0x010, /* shall be created */ FIL_TRUNC = 0x020, /* shall be truncated */ FIL_SYNC = 0x040, /* syncing output */ /* lock flags */ FIL_TLOCK = 0x100, /* try locking (EX with WR) */ FIL_BLOCK = 0x200, /* blocking lock (EX with WR) */ FIL_FLOCK = 0x300 /* any locking is set */ }; /** open a new fid based on name and flags. TLOCK can be specified on any plattform, translating to a fcntl full file lock on *nix and a share mode on win @return 0 or some error code */ extern int fOpen ( file *f, const char *name, int flags ); extern int fClose ( file *f ); extern int fSize ( file f ); extern unsigned fTime ( file f ); /* mtime sec */ /* Like the syscalls, this returns the number of bytes on success, 0 on eof. fPwrite repeats and does not return an error when interrupted. On error, a negative value is returned. */ extern int fRead ( file *f, void *buf, unsigned len ); extern int fWrite ( file *f, const void *buf, unsigned len ); extern int fPread ( file *f, void *buf, unsigned len, unsigned off ); extern int fPwrite ( file *f, const void *buf, unsigned len, unsigned off ); extern int fSeek ( file *f, unsigned off ); extern int fTrunc ( file *f, unsigned length ); #ifdef BUILD_SHMODE /* remove a full file lock as set by fOpen */ extern void fUnlock ( file f ); /* lock byte n use TLOCK or BLOCK, possibly with WR, to lock, 0 to unlock. */ extern int fLock ( file f, unsigned n, int flg ); #define FLOCK(f,n,flg) (ENV_SHARED==env.wri && fLock(f,n,flg)) #else #define FLOCK(f,n,flg) 0 #endif #define FLOCKSH(f,n) FLOCK(f,n,FIL_BLOCK) #define FLOCKEX(f,n) FLOCK(f,n,FIL_BLOCK|FIL_WR) #define FLOCKUN(f,n) FLOCK(f,n,0) /** slurp in a whole file at once. @param buf points to buffer of size sz. lio_slurp will allocate one, if *buf is NULL @param sz maximum number of bytes to read @param name of file to slurp @param opt if != 0, do not complain on failure @return number of bytes read or negative on error */ extern int fSlurp ( char **buf, int sz, const char *name, int opt ); /* record-oriented sink to (file)dst. Expects the list to contain a proper record. If the header starts with a digit, a W is prepended. An empty header is ommited. */ extern void fSinkr (Sink *that, int eor); /* line-oriented sink (field values only) to (file)dst. leaves the header alone, ignores tags and blank values. yet prints a blank line on eor. */ extern void fSinkl (Sink *that, int eor); /** a potentially mapped file */ typedef struct { file fil; int flg; char *map; unsigned npg; /* in pages of env.psz */ unsigned lim; /* max pages to map */ #ifdef WIN32 char *nam; /* for shared mapping */ void *hdl; /* "mapping object" */ #endif } FMap; /** open a file to be mapped. Like fOpen, but saves flags for later reference. */ extern int fMOpen ( FMap *fm, const char *name, int flags ); extern int fMClose ( FMap *fm ); /** map, remap or unmap a memory mapping @param fm the filemap members fil and flg must be set as of fOpen. if fm was mapped, the existing mapping is unmapped (or remapped) @param npg number of pages to map; 0 for no new mapping @return mapped length; if <= 0, fm->map is set to 0, else to memory region */ extern int fMap ( FMap *fm, unsigned npg ); /** sync a mapped page */ extern int fMSync ( FMap *fm, unsigned page ); /* file input buffer structure for fGets. suitable for both temp fix buffers and, with some care, for List buffers. */ typedef struct { file f; /* file to read from */ unsigned n; /* current line number (1 based) */ unsigned o; /* offset of p from file start */ char *b; /* buffer base (const) */ unsigned s; /* buffer size (const) */ char *p; /* start of current line */ unsigned l; /* line len */ unsigned m; /* more bytes after p+l (including the LF) */ } FBuf; /* sloppy but convenient initializer macro */ #define FIL_BUF( fb, fil, buf ) do { \ fb.f = fil; fb.b = fb.p = buf; fb.s = sizeof(buf); \ fb.n = 1; fb.o = fb.l = fb.m = 0; \ } while (0) /* even more convenient macro, must be end of data def */ #define FIL_DEFBUF( f ) \ char buf[0x2000]; \ FBuf fb; \ FIL_BUF(fb, f, buf); /* set p and l to next line, lines are terminated by LF. if l, advance p after current line. if m, skip next byte. if m, search for newline. if no newline found, mv p downto b and read more chars upto s. if l==s, buffer is exhausted (and m is 0). else if m, p[l] is a newline. else if FIL_NONE == f, no bytes could be read (an eof was seen or somebody set NBLOCK). return whether we got a line */ extern int fGets ( FBuf *fb ); extern int fGetr ( List *l, FBuf *fb ); /* ****************************************************************** */ /* */ /* DATABASE */ /* */ /* ****************************************************************** */ enum { /* flags for record and query data/index file pairs */ DX_OPEN = 0x1, /* open */ DX_WRITE = 0x2, /* open for writing */ DX_ASYNC = 0x4, /* no synced write */ DX_MODIF = 0x8 /* modified */ }; /** record data and index. While this is accessible standalone, a logical database table may consist of several Rdx, e.g. per every million records. */ typedef struct Rdx { file mrd; FMap mrx; int flg; int mid; /* in records == maxid */ int rdl; /* length of data file in bytes */ int ptl; /* pointer bytes, by now always 8 */ int typ; /* type of pointer file */ } Rdx; /** initialise from an already open fd. */ extern int rInit ( Rdx *rdx ); /** flush and release any cache. */ extern void rFini ( Rdx *rdx ); /* read record rid. if mpos and the rec is found at mpos or higher, rRead will backtrack to earlier versions. */ extern int rRead ( List *l, Rdx *rdx, int rid, unsigned mpos ); /* write a record use rid 0 to get new rid use opos -1 if you don't care about old pos to be transparent here, the rid@pos as found in standard recs is ignored and should be pre-skipped up to the leader. specify the record size if known, with 0, rSiz will be used @return the record id written (> 0) on success, <= 0 on error */ extern int rWrite ( Rdx *rdx, const Fld *r, int rid, int opos, unsigned siz ); /** query data and index leaf blocks have a configurable size from 512 bytes to 8K fork (inner node) blocks have always pagesize, i.e. 4K up to 64K While this is accessible standalone, a logical database table may consist of several Qdx, each holding some range of keys. */ typedef struct Cdx Cdx; typedef struct Qdx { /* actually it's a B-Link-Tree */ file mqd; /* the leaves file */ FMap mqx; /* the tree file */ const Cdx *cdx; /* the collation */ int flg; /* flags: writeable */ unsigned char typ; /* cfg: leaf block type */ unsigned char ksz; /* cfg: max key length, default 255 */ unsigned char ptr; /* cfg: inverted file pointer type or plain value size */ unsigned char let; /* cfg: pct free on load */ /* members set automatically: */ unsigned char vsz; /* value size, min 4, default 8 */ unsigned char uni; /* value unique length (see qSet) */ unsigned char ftp; /* fork block type */ unsigned char dpt; /* depth (level of root over bottom > 0) */ unsigned lsz; /* leaf block size computed from type */ unsigned lln; /* # leaf blocks in index */ unsigned fln; /* # fork blocks in index */ /* members considered internal: */ struct QLoad *qld; } Qdx; enum { /* btree block type, size, flg */ QDX_TYPMSK = 0xC0, /* highest 2 bits: basic type */ QDX_LEAF = 0x00, /* leaf block, portable */ QDX_FORKLE = 0x40, /* fork block little endian */ QDX_FORKBE = 0x80, /* fork block big endian */ QDX_LEAFPV = 0xC0, /* leaf plain values (forks don't care) */ /* next 2 bits 0x30 for future extensions */ QDX_COMPRS = 0x08, /* flag compressed keys (not yet supported) */ QDX_SIZMSK = 0x07, /* lowest 3 bits: blocksize */ QDX_LEAF0K = 0x00, /* 1/2K blocks 0x0200 9+0 bits */ QDX_LEAF1K = 0x01, /* 1K blocks 0x0400 */ QDX_LEAF2K = 0x02, /* 2K blocks 0x0800 */ QDX_LEAF4K = 0x03, /* 4K blocks 0x1000 */ QDX_LEAF8K = 0x04 /* 8K blocks 0x2000, max for leaves */ }; enum { QDX_MAXVALPERLEAF = 0x800, /* max 8K / min 4 bytes vsz */ QDX_LEAFSH = 9, /* leaf size shift+(0..4) ~ 512 bytes - 8K */ QDX_FORKSH = 12 /* fork size shift+(0..4) ~ 4K - 64K */ }; typedef struct { unsigned char len; unsigned char byt[255]; } Val; typedef struct { Val val; unsigned char len; unsigned char byt[255]; } Key; typedef struct QLoop QLoop; /** callback for index loop. maybe called multiple times for same key, if it spans blocks. In this case, flag QSAME is set. loop stops if QCb returns != 0 */ typedef int QCb ( QLoop *self ); struct QLoop { QCb *qcb; Qdx *qdx; int flg; Key key; Key to; /* set on callback: */ Key cur; unsigned nvals; const unsigned char *vals; }; enum { /* flags */ QLOOP = 0, /* loop endless */ /* stop based on QLoop.key */ QEQ = 1, /* loop while == key */ QPF = 2, /* loop on prefix key */ /* stop based on QLoop.to */ QUPTO = 4, /* loop while < to */ QINCL = 5, /* loop while <= to */ QSTOP = 7, /* mask for stop mode */ QSKIP = 8, /* skip the from key */ QSAME = 0x10 /* callback on same key */ }; /** initialise from an already open fd. */ extern int qInit ( Qdx *bt ); /** flush and release any cache. */ extern void qFini ( Qdx *bt ); /* load a sorted series of keys and hits into index. call repeatedly, using a key with val.len 0 in last call */ extern int qLoad ( Qdx *bt, Key *key ); extern int qLoadf ( Qdx *bt, file *f ); /* write the key-value pair to the index and return 0, unless: - the value is zero on the 1st uni bytes and there is already such a value (unique key, return 3) - there is an all-zero value for the key (stopword, return 2) - it is already there (full duplicate found, return 1) With plain values, only full duplicates are checked. Uni is usually the length of the initial segment of value, which is the rid in fulltext mode, else the tag. Where a unique key is found, the value is copied to key. */ extern int qSet ( Qdx *bt, Key *key ); enum { QST_OK, QST_FOUND, QST_STPWRD, QST_UNIKEY }; /* delete a key-value pair. return 1 if found, else 0 */ extern int qDel ( Qdx *bt, Key *key ); extern int qLoop ( QLoop *self ); /* standard values structured as pointers denoting a record and position where key occurred. The value structure is up to 3 big endian unsigned numbers: 0..2 bytes for tag, 3+(0..3) bytes for rid (in fulltext mode: before the tag) and 0..4 bytes for pos, totalling from 4 (vsz min.) to 12 bytes. */ typedef struct { /* where a key has a hit */ unsigned short tag; /* while neg. tags are allowed, sorting is unsigned */ unsigned short ext; /* extend row id to six bytes (used as db number) */ unsigned rid; /* row id */ unsigned pos; /* word pos, usually with field occ<<16 */ } Ptr; enum { QDX_TAGMSK = 0xC0, /* mask length of tag 0..2 */ QDX_TAG1 = 0x40, QDX_TAG2 = 0x80, QDX_TAGSH = 6, /* ... shifted by 6 */ QDX_RIDMSK = 0x30, /* mask length of rid 0..3 */ QDX_RIDMIN = 3, /* based on 3 */ QDX_RIDSH = 4, /* ... shifted by 4 */ QDX_RID3 = 0x00, QDX_RID4 = 0x10, QDX_RID5 = 0x20, QDX_RID6 = 0x30, QDX_FULTXT = 0x08, /* traditional fulltext ordering */ QDX_POSMSK = 0x07, /* mask length pos info 0..4 */ /* default settings based on 3 byte rid */ QDX_ISIS = QDX_RID3|QDX_TAG2|QDX_FULTXT|3, /* 0x8B, 8 byte fulltext 3+2+3 IFP format */ QDX_STDDB = QDX_TAG1|QDX_RID3 /* 0x40, 4 byte 3 rid + 1 for 256 field tags */ }; /* decode a value to a pointer */ extern void qRdVal ( Ptr *ptr, const unsigned char *val, unsigned char typ ); /* create a value from hit. */ extern void qMkVal ( Val *val, Ptr *ptr, unsigned char typ ); /* decode a key to plaintext */ extern int qRdKey ( Qdx *qdx, char *plain, int l, Key *key ); /* create a key from plaintext, truncating to ksz. */ extern void qMkKey ( Qdx *qdx, Key *key, char *b, int l ); /* create a key and value from line. */ extern int qMkKeyVal ( Qdx *qdx, Key *key, char *b, int l ); typedef struct { Qdx *qdx; int del; Key pfx; Ptr ptr; } QSet; /* set or del one keys basically this behaves like qMkKey, qMkVal, qSet ptr.pos is incremented with pfx, pfx is prepended to the key (before qMkKey) */ extern int qSetKeyVal (QSet *qst, char *val, int len); /* split the value into words and qSetKeyVal each return the number of entries made */ extern int qSetKeyVals (QSet *qst, char *val, int len); /** collation */ #define CDX_MAXSEQ 15 /* max byte sequence length */ extern const Cdx *cOpen ( const Fld *src ); extern int cEnc ( const Cdx *cdx, Key *key, unsigned char *b, int l, int w ); extern int cDec ( const Cdx *cdx, unsigned char *b, int l, Key *key ); enum { /* see Metadata.txt */ MET_OPT = 001, MET_UNU = 002, MET_CTP = 003, MET_COL = 004, MET_VER = 005, MET_FLD = 006 }; /** finally, a real database */ typedef struct Db { char *nam; char *pat; int flg; int mnt; /* mount count */ struct Db *nxt; /* linked list */ const Fld *opt; /* inner meta data */ Rdx rdx; Qdx qdx; } Db; extern Db *dOpen (const char *dbname); extern void dClose (Db *db); extern void dCloseAll (); /** access to record and query data via the db. In future versions these might become real functions taking care of multifile databases. */ #define dRead(l,db,rid) ((rid)?rRead(l,&(db)->rdx,rid,0):lCpy(l,(db)->opt,0)) #define dWrite(db,r,rid) rWrite(&(db)->rdx,r,rid,-1,0) #define dSet(db,key) qSet(&(db)->qdx,key) #define dDel(db,key) qDel(&(db)->qdx,key) #define dLoop(db,ql) ((ql)->qdx=&(db)->qdx, qLoop(ql)) #define CORE_H #endif /* CORE_H */ mlt/db.c0100644000076400001440000001623410175233623011241 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: db.c,v 1.1 2005/01/24 17:59:47 kripke Exp $ implementation of general db access functions. */ #include "core.h" /* ************************************************************ private types */ /** file extensions not listed on filext.com */ /** B-Link-Tree malete query data/index */ static const char EXT_MQD[] = ".mqd"; static const char EXT_MQX[] = ".mqx"; /** plaintext malete record data/index */ static const char EXT_MRD[] = ".mrd"; static const char EXT_MRX[] = ".mrx"; /** record 0 options data */ static const char EXT_M0D[] = ".m0d"; /* ************************************************************ data */ /* linked list of open dbs. */ static Db *dbs; enum { /* additional flg outside the short range */ OPEN_TRY = 0x10000, /* try writable, open readonly else */ OPEN_UC = 0x20000, /* use uppercase ext */ /* commonly used combinations */ /* 1) open as is, do not complain about any failure, do not create */ OPEN_ASIS = FIL_RDWR|OPEN_TRY, /* 2) open readonly, do not complain about any failure, do not create */ OPEN_RDIF = FIL_RD|FIL_TRY, /* 3) open readonly, complain about any failure */ OPEN_RD = FIL_RD, /* 4) open or create writable, complain on failure */ OPEN_NEW = FIL_RDWR|FIL_CREAT, OPEN_BLANK = FIL_RDWR|FIL_CREAT|FIL_TRUNC }; /* set extension. fname MUST already end with .xxx. */ static char *setext ( char *fname, const char *ext ) { int l = strlen( fname ) - 4; memcpy( fname+l, ext, 4 ); return fname; } Db *dOpen (const char *dbname) { const Fld *cfg; Fld opt; file optf; /* options during open */ int withrdx = !0; int withqdx = !0; int filmode = FIL_RD, async = DX_ASYNC; /* changed by -w */ int lck = FIL_TLOCK; /* set to BLOCK by -b */ /* misc vars */ Db *db; int plen, ret = 0; #ifdef BUILD_SHMODE if (ENV_SHARED==env.wri) async = 0; /* default to sync */ #endif for ( db = dbs; db; db = db->nxt ) if ( !strcmp(db->nam, dbname) ) { if ( DX_OPEN & db->flg ) { db->mnt++; return db; } return 0; /* hmmm ... should not be here */ } cfg = rKey(env.opt->fld, 1, dbname); db = mAlloc(sizeof(Db)); db->rdx.mrd = db->rdx.mrx.fil = FIL_NONE; db->rdx.mrx.lim = env.rml; db->qdx.mqd = db->qdx.mqx.fil = FIL_NONE; db->qdx.mqx.lim = env.qml; plen = strlen(dbname); memcpy(db->nam = mAlloc(plen+1), dbname, plen+1); /* pre-options defaults */ db->qdx.typ = QDX_LEAF1K; db->qdx.ptr = QDX_ISIS; db->qdx.ksz = 255; if ( env.wri ) db->flg |= DX_WRITE; if ( cfg ) for ( opt.val = 0; vGet(&opt, cfg, "bfqrw"); ) switch (opt.tag) { case 'b': /* blocking */ lck = FIL_BLOCK; continue; case 'f': /* file/path */ if ( !opt.len ) { eRr(LOG_WARN, "bad empty file"); continue; } db->pat = mAlloc(opt.len + plen + 5); memcpy(db->pat, opt.val, opt.len); switch ( opt.val[opt.len - 1] ) { case '/': #ifdef WIN32 case '\\': #endif memcpy(db->pat+opt.len, dbname, plen); plen += opt.len; break; default: plen = opt.len; } eRr(LOG_INFO, "using path %.*s", plen, db->pat); continue; case 'q': /* qdx type */ if ( !opt.len ) withqdx = 0; else { unsigned char val = (unsigned char) a2i(opt.val+1, opt.len-1); switch (*opt.val) { case 'k': db->qdx.ksz = val; break; case 'l': db->qdx.let = val; break; case 'p': db->qdx.ptr = val; break; case 't': db->qdx.typ = val; break; } } continue; case 'r': /* mrx type */ if ( !opt.len ) withrdx = 0; else db->rdx.typ = a2i(opt.val, opt.len); continue; case 'w': /* writable */ if ( opt.len ) switch ( *opt.val ) { case '0': db->flg &= ~DX_WRITE; break; case 'a': async = DX_ASYNC; break; case 's': async = 0; break; } continue; } /* post-options defaults */ if ( DX_WRITE & db->flg ) { db->rdx.flg |= DX_WRITE; db->qdx.flg |= DX_WRITE; filmode |= FIL_WR|FIL_CREAT; if ( !(DX_ASYNC & (db->flg |= async)) ) filmode |= FIL_SYNC; } if ( !db->pat ) memcpy(db->pat = mAlloc(plen+5), dbname, plen); memcpy(db->pat+plen, ".???", 5); #if 1 /* check options file */ if ( ! fOpen(&optf, setext(db->pat,EXT_M0D), FIL_RD|FIL_TRY) ) { char buf[0x10000]; int sz; if ( 0 < (sz = fSize(optf)) ) { char *p = sz < (int)sizeof(buf) ? buf : mAlloc(sz); eRr(LOG_INFO, "reading %d bytes options from '%s'", sz, db->pat); if ( sz == fRead(&optf, p, sz) ) { List dbopt; lParse(lInit(&dbopt,0), p, sz); db->opt = LCAN(&dbopt); } if ( buf != p ) mFree(p); } fClose(&optf); } #endif #ifdef BUILD_SHMODE if (ENV_SHARED==env.wri) lck = 0; /* we lock on demand */ #endif #define CLEANUP( args ) do { ret = eRr args; goto cleanup; } while (0) if ( withrdx ) { if ( fOpen( &db->rdx.mrd, setext(db->pat,EXT_MRD), filmode|lck ) ) CLEANUP((LOG_ERROR, "no master file '%s'", db->pat )); if ( fMOpen( &db->rdx.mrx, setext(db->pat,EXT_MRX), filmode ) ) CLEANUP((LOG_ERROR, "no pointer file '%s'", db->pat )); if ( rInit( &db->rdx ) ) CLEANUP((LOG_ERROR, "could not rInit '%s'", db->pat )); } if ( withqdx ) { if (db->opt) db->qdx.cdx = cOpen(db->opt); if ( fOpen( &db->qdx.mqd, setext(db->pat,EXT_MQD), filmode|lck ) ) CLEANUP((LOG_ERROR, "no index leaves file '%s'", db->pat )); if ( fMOpen( &db->qdx.mqx, setext(db->pat,EXT_MQX), filmode ) ) CLEANUP((LOG_ERROR, "no index tree file '%s'", db->pat )); if ( qInit( &db->qdx ) ) CLEANUP((LOG_ERROR, "could not qInit '%s'", db->pat )); } db->pat[plen] = 0; /* cut .ext */ db->flg |= DX_OPEN; db->nxt = dbs; dbs = db; return db; cleanup: /* cleanup ... */ db->flg = 0; fClose( &db->qdx.mqd ); fMClose( &db->qdx.mqx ); fClose( &db->rdx.mrd ); fMClose( &db->rdx.mrx ); return 0; } /* dOpen */ void dClose ( Db *db ) { Db **d; for ( d = &dbs; *d; d = &(*d)->nxt ) /* unlink */ if ( db == *d ) { if ( --db->mnt ) return; *d = db->nxt; break; } if ( FIL_NONE != db->rdx.mrd ) { rFini( &db->rdx ); fClose( &db->rdx.mrd ); fMClose( &db->rdx.mrx ); } if ( FIL_NONE != db->qdx.mqd ) { qFini( &db->qdx ); fClose( &db->qdx.mqd ); fMClose( &db->qdx.mqx ); } db->flg = 0L; mFree( (void*)db->pat ); mFree( (void*)db->opt ); mFree( (void*)db->nam ); memset( db, 0, sizeof(db) ); mFree( db ); } /* dClose */ void dCloseAll () { Db *d, *nxt; for ( d = dbs; d; d = nxt ) { nxt = d->nxt; dClose(d); } } /* dCloseAll */ mlt/dbo.c0100644000076400001440000011477210233541566011431 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: dbo.c,v 1.2 2005/04/26 22:40:54 kripke Exp $ the db object */ #include "pw.h" #define Q_MAX_EXP 500 #define Q_MAX_DEPTH 50 enum { /* nodes in a query expression */ OP_NULL, OP_HEAD, /* blank node and head pseudo operator */ /* operators in increasing precedence; level is op>>1 */ OP_OR, OP_AND=4, OP_NOT, /* AND and NOT/WITHOUT */ OP_TAGS, /* /(...) tagmask, "kind of" unary */ OP_SAMEREP = 8, OP_SAMETAG, /* ,(F) ;(G) same field repetition/tag */ OP_DISTMAX, OP_DISTEX, /* ... and $$$ max/ex distance */ OP_IDENT, /* last operator, really first simple expression */ T_QRF = 16, /* #n ref */ /* term and relation */ T_EQ, T_PFX, /* term equality and prefix match */ T_GT, T_GTE, /* relational */ T_LT, T_LTE, /* relational */ T_CONT, T_RE /* contains and regexp */ }; static const char *opname[] = { "null", "head", "+", "", "*", "^", "/", "", ",", ";", ".", "$", "-" }; static const char *tname[] = { "#", "=", "%", ">", ">=", "<", "<=", ":", "~" }; static const int opdim[] = { 0, 0, 0, 0, 0, 0, 1, 0, /* / pseudo op */ 2, 1, /* , ; */ 3, 3, /* . $ */ 3 /* IDENT does not actually imply a dimension */ }; typedef union Exp Exp; /* query expression */ typedef struct Bop { /* binary operator */ int tag; int dst; /* distance */ Exp *lft; /* index of left ... */ Exp *rgt; /* ... and right operand */ } Bop; typedef struct Tfl { /* tag filter */ int tag; int len; /* # tags */ Exp *lft; /* index of left expression, MUST match pos of Bop.lft */ unsigned short *lst; /* taglist */ } Tfl; typedef struct Qrf { /* query ref */ int tag; /* == T_QRF */ Qry *qry; } Qrf; typedef Fld Trm; union Exp { /* query expression */ int tag; Bop bop; Tfl tfl; Qrf qrf; Trm trm; }; struct Qry { /* parsed query and search result */ Qry *nxt; /* chained */ Dbo *dbo; /* our database */ lolo use; /* req of last use */ unsigned qid; /* our id */ char *txt; /* full unparsed expression */ unsigned sln; /* length of search expression, txt[sln] is '?' or 0 */ unsigned cur; /* sor */ unsigned len; /* of result set */ lulu *res; /* result set, 0 for all */ lulu skp; /* min rid to consider (skip to) */ lulu brk; /* min rid where set size limit was hit */ Tfl srt; /* sort filter */ Exp *exp; /* parsed search or filter expression */ char *buf; /* holding terms and taglists */ Exp ebf[Q_MAX_EXP]; /* exp buffer */ }; /* qry->exp is either 0 or ebf: after parsing the search part, it is ebf, possibly the OP_NULL expression. Then it is set to 0, if there is no '?' (no filter: record ids only), or to ebf for a filter expression (OP_NULL selecting all). */ typedef struct Prv { /* dbo private data */ Dbo dbo; /* public */ Qry qry; /* the null query */ } Prv; typedef struct { /* level in tree during parsing */ Exp *exp; int par; /* number of open parentheses */ } Lev; /* see doc/Query.txt */ #define EXP_ISSIMPLE(e) (OP_IDENT <= (e)->tag) static int canmark ( Exp *e ) { return EXP_ISSIMPLE(e) ? 1 : (OP_OR == e->tag /*|| OP_NOT == e->tag*/) ? canmark(e->bop.lft) && EXP_ISSIMPLE(e->bop.rgt) : OP_TAGS == e->tag ? canmark(e->tfl.lft) : 0; } /* mode is 'c'reate, 'm'ark or 'f'ilter dim the required dimension return number of buffers required */ static int explain (Exp *e, char mode, int dim, unsigned short tag) { char lmode, rmode; int lcost, rcost, cost; if ( T_QRF <= e->tag ) { if ( T_QRF < e->tag ) eRr(LOG_VERBOSE, "%c%d %s \"%.*s\"/%d", mode, dim, tname[e->tag-T_QRF], e->trm.len, e->trm.val, tag); else if ( tag || dim ) { eRr(ERR_INVAL, "tag %d dim %d at ref#%d", tag, dim, e->qrf.qry->qid); return -1; } else eRr(LOG_VERBOSE, "%c%d #%d", mode, dim, e->qrf.qry->qid); return 'm' == mode ? 0 : 1; } if ( OP_TAGS == e->tag ) { eRr(LOG_VERBOSE, "/(%d,..)[%d]", *e->tfl.lst, e->tfl.len); return e->tfl.lft ? explain(e->tfl.lft, mode, dim, *e->tfl.lst) : 0; } if ( OP_DISTMAX == (e->tag&~1) ) eRr(LOG_VERBOSE, "%c%d %s%d", mode, dim, opname[e->tag], e->bop.dst); else eRr(LOG_VERBOSE, "%c%d %s", mode, dim, opname[e->tag]); if ( !e->tag ) return 0; /* bin op */ lmode = rmode = mode; cost = 0; if ( OP_OR == e->tag ) { if ( 'f' == mode ) cost = 1; /* because we need to keep the filter */ } else if (canmark(e->bop.rgt)) { rmode = 'm'; } else { lmode = 'f'; if ( 'f' == mode ) cost = 1; /* double filtering */ } if ( dim < opdim[e->tag] ) dim = opdim[e->tag]; lcost = e->bop.lft ? explain(e->bop.lft, lmode, dim, tag) : 0; if ( 0 > lcost ) return lcost; rcost = e->bop.rgt ? explain(e->bop.rgt, rmode, dim, tag) : 0; if ( 0 > rcost ) return rcost; cost += lcost > rcost ? lcost : rcost; return cost; } /* explain */ /* parse search or filter expression tokens are appended to an array of expressions. Operators form a tree in this array, with 0 being the head. We always operate on right childs, which, however, may become the left child of following operators. We maintain a path of right childs up to the current node. The current node can either be blank or a closed expression (after a term or closing parenthesis). If the next token is - an operator, the current node must not be blank for most operators (although some may use a default expression as left hand). While nodes on the path have same or higher precedence than the operator, and no node with open parentheses is encountered, the path is unwound. The then current node becomes the operators left child and is replaced by the operator. The operator opens a blank node as right child. - a terminal (term or ref), if the current node is not blank, a default AND is inserted. Then we have a blank node, which becomes the terminal. - a left parenthesis also produces a default AND, if the current node is not blank. Then the blank nodes parentheses count is increased. - a right parenthesis behaves like an operator in that the current node must not be blank. It then unwinds the path up to the first node whose parentheses count is not zero and decreases it. buffer usage: query length is sufficient. - quoted terms, without the quotes, which is 2 bytes less than in the query expression. may run on the 0 byte in unterminated literal, however, the inital quote accounts for that. - 2 bytes per tag in tagfilter. since every tag is at least 1 digit plus 1 extra character (the / or terminating ,), needs not more bytes then in the query expression. we may have to align to 2 after a proper quoted term, but the terms closing quote accounts for that. This may be called twice: - on the full query text, with q->exp unset and q->txt == txt - after the ?, reusing q->exp */ static int parse (Qry *q, unsigned char *txt) { Lev path[Q_MAX_DEPTH], *cur = path; Exp *exp; int n/*# used nodes*/, cost; unsigned char *buf = (unsigned char *)q->buf + q->sln; /* buffer might still be used by q->srt */ unsigned char *p = txt, *pos=p; eRr(LOG_VERBOSE, "parsing '%s' as %s", txt, q->exp ? "filter" : "search"); if ( q->exp ) /* in filtering -- reuse exp buffer */ memset(exp = q->exp, 0, Q_MAX_EXP * sizeof(Exp)); else exp = q->ebf; memset(path, 0, sizeof(path)); /* add the pseudo op head with lowest precedence */ cur->exp = exp; exp->tag = OP_HEAD; exp->bop.rgt = (++cur)->exp = exp+1; /* add blank */ n = 2; #define QRAP( args ) do { eRr args; goto qrap; } while (0) /* query is crap */ for (;;) { Exp tok; /* get a token */ while (33 > *p) if ( !*p++ ) { p--; goto done; } /* skip white */ pos = p; LOG_DBG(LOG_DEBUG, "pos %d lev %d: %c", pos-txt, cur-path, *p); switch (*p) { case '+': tok.tag = OP_OR; p++; break; case '*': tok.tag = OP_AND; p++; break; case '^': tok.tag = OP_NOT; p++; break; case ';': tok.tag = OP_SAMETAG; p++; break; case ',': tok.tag = OP_SAMEREP; p++; break; case '.': tok.tag = OP_DISTMAX; for ( tok.bop.dst=1; '.' == *++p; ) tok.bop.dst++; break; case '$': tok.tag = OP_DISTEX; for ( tok.bop.dst=1; '$' == *++p; ) tok.bop.dst++; break; case '-': tok.tag = OP_IDENT; p++; break; case ')': eRr(LOG_VERBOSE, ") on lev %d", cur-path); if ( !cur->exp->tag ) QRAP((ERR_INVAL,"unexpected )")); while ( !cur->par ) if ( path == --cur ) QRAP((ERR_INVAL,"unbalanced )")); cur->par--; eRr(LOG_VERBOSE, "now on lev %d", cur-path); p++; continue; case '/': { int multi = '(' == *++p; unsigned short *lst; if ( multi ) p++; tok.tag = OP_TAGS; if ( 1 & (int)buf ) buf++; /* align */ lst = tok.tfl.lst = (unsigned short*)buf; for (;;) { int tag, got; while (*p && 33 > *p) p++; /* skip white */ got = a2il((char*)p, -1, &tag); if ( !got ) QRAP((ERR_INVAL, "/ expects digit")); if ( tag<0 || 65535 *p) p++; /* skip white */ if ( ')' == *p++ ) break; if ( ',' != p[-1] ) QRAP((ERR_INVAL, "/ expects ,")); } tok.tfl.len = lst - tok.tfl.lst; buf = (unsigned char*)lst; if ( !cur->exp->tag && 2==n ) { /* save special case */ tok.tfl.lft = 0; if ( q->exp ) /* parsing filter */ *(exp->bop.lft = exp + n++) = tok; /* as left of head */ else q->srt = tok.tfl; /* as sort expression */ continue; } break; } /* tags */ case '?': if ( q->exp ) QRAP((ERR_INVAL, "unexpected ? in filter")); goto done; case '(': if ( ('F' == p[1] || 'G' == p[1]) && ')' == p[2] ) { /* legacy ops */ tok.tag = 'F' == p[1] ? OP_SAMEREP : OP_SAMETAG; p += 3; break; } if ( CT_D == lat1ct[p[1]] ) { /* (n) */ int dst, dig = a2il((char*)p+1,-1,&dst); if ( ')' == p[1+dig] ) { tok.tag = OP_DISTMAX; tok.bop.dst = dst; p += dig+2; break; } } default: /* terminal or parenthesis */ if ( cur->exp->tag ) { /* add default AND */ tok.tag = OP_AND; break; } if ( '(' == *p ) { eRr(LOG_VERBOSE, "( on lev %d", cur-path); cur->par++; p++; continue; } #ifdef REFS_ARE_NOW_FIXED if ( '#' == *p ) { unsigned qid; Qry *qq; if ( CT_D != lat1ct[*++p] ) QRAP((ERR_IDIOT,"#expects digit")); p += a2il((char*)p, -1, (int*)&qid); for (qq = q->dbo->qry; qq && qid != qq->qid; qq = qq->nxt) ; if ( !qq ) QRAP((ERR_BADF, "query #%u not found (maybe expired)", qid)); if ( q->exp ) QRAP((ERR_IDIOT, "sorry, refs supported in search only")); tok.tag = T_QRF; tok.qrf.qry = qq; *cur->exp = tok; continue; } #endif /* check for relation */ tok.tag = T_EQ; switch (*p) { case '=': p++; break; case '%': tok.tag = T_PFX; p++; break; case '>': if ( '=' != *++p ) tok.tag = T_GT; else { tok.tag = T_GTE; p++; } break; case '<': if ( '=' != *++p ) tok.tag = T_LT; else { tok.tag = T_LTE; p++; } break; case ':': if ( !q->exp ) QRAP((ERR_INVAL, ": only in filter")); tok.tag = T_CONT; p++; break; case '~': /* tok.tag = T_RE; p++; break; */ QRAP((ERR_INVAL, "~ unsupported")); } while (33 > *p) if ( !*p++ ) goto done; /* skip white */ if ( '"' != *p ) { /* plain - point to expr */ if ( CT_S == lat1ct[*p] ) QRAP((ERR_INVAL,"bad character '%c'",*p)); tok.trm.val = (char*)p; while (!((CT_WHITE|CT_SPECL) & lat1ct[*p++])) ; tok.trm.len = (char*)--p - tok.trm.val; } else { /* quoted - use buffer */ tok.trm.val = (char*)buf--; while ((*++buf = *++p) && ('"'!=*buf || '"'==*++p)) ; if ( !*buf ) QRAP((ERR_INVAL,"unexpected end of query in literal")); tok.trm.len = (char*)buf - tok.trm.val; } if ( '$' == *p ) { tok.tag = T_PFX; p++; } *cur->exp = tok; continue; } /* not continued: got operator */ eRr(LOG_VERBOSE, "%s on lev %d", opname[tok.tag], cur-path); if ( !cur->exp->tag ) QRAP((ERR_INVAL,"unexpected operator '%s'", opname[tok.tag])); if ( OP_DISTMAX > tok.tag ) /* the DISTs are right associative with highest precedence */ /* for all others, unwind path up to lower precedence or parenthesis */ while ( !cur->par ) if ( ((--cur)->exp->tag&~1) < (tok.tag&~1) ) { cur++; break; } /* now cur->exp is the victim to become our left child */ tok.bop.lft = cur->exp; cur->exp = cur[-1].exp->bop.rgt = exp + n++; /* get new op exp */ if ( n > Q_MAX_EXP-1 ) /* allow one for right child */ QRAP((ERR_INVAL, "max #expressions %d exceeded", Q_MAX_EXP)); *cur->exp = tok; if ( OP_TAGS != tok.tag ) { Exp *child = cur->exp->bop.rgt = exp + n++; /* empty right child */ if ( (++cur - path) == Q_MAX_DEPTH ) QRAP((ERR_INVAL, "max depth %d exceeded", Q_MAX_DEPTH)); cur->exp = child; cur->par = 0; } } /* for ever */ done: if ( !cur->exp->tag && n>(exp->bop.lft ? 3 : 2) ) /* ok only if completely empty */ QRAP((ERR_INVAL,"unexpected end of query after operator")); /* to be strict, we should check for open parenthesis -- but who cares */ if ( !exp->bop.lft ) /* purge head unless we have a tagfilter */ *exp = *exp->bop.rgt; cost = explain(exp,'c',0,0); if ( 0 > cost ) QRAP((ERR_INVAL,"no plan")); eRr(LOG_VERBOSE, "successfully parsed %d bytes cost %d", p - txt, cost); q->exp = exp; return p - txt; qrap: /* cleanup */ #undef QRAP eRr(ERR_INVAL, "error in query '%s' near pos %d", txt, pos-txt); q->exp = 0; return ERR_INVAL; } /* parse */ typedef struct { /* long long pointer */ lulu rit; /* higher 6 rid, lower two tag */ unsigned pos; } Llp; #define RIDMSK LULU(0xffffffffffff0000) #define OCCMSK 0xffff0000U #define PGT(a, b) \ ((a)->rit > (b)->rit ? 1 : (a)->rit < (b)->rit ? 0 : (a)->pos > (b)->pos) typedef struct Res { int len; Llp ptr[QDX_MAXVALPERLEAF]; /* really ses->s */ } Res; /* followed by ses->s marks */ #define RES_MARKS(res) (((char*)(res))+4+ses->s*sizeof(Ptr)) /* TODO: maintain pool */ static Res *rAlloc () { return mAlloc(4 + ses->s*(sizeof(Ptr)+1)); } static void rFree (Res *res) { mFree(res); } typedef struct { /* a filter */ Res *res; /* results against which to filter */ Bop *cnd; /* matching condition */ char *mrk; /* optional mark array */ char set; /* the mark to set/ positive or negative filtering */ } Flt; /* match res against flt on cnd with mrk, set mrk to set for corresponding entries in flt else compact res, keeping matches is set, nonmatches else TODO: prolly we'd rather use a copy of flt; however, compiler might do this */ static void rMatch (Res *res, Flt *flt) { Llp *p = res->ptr, *pe = p + res->len, *d = p; Llp *f = flt->res->ptr, *fe = f + flt->res->len; unsigned dim = opdim[flt->cnd->tag], pos; /* filter dim, pos */ char *m; #ifndef NDEBUG int mcnt = 0; # define MCNT mcnt++; #else # define MCNT #endif eRr(LOG_VERBOSE, "matching res len %d against flt len %d cnd %d set %d mrk %d dim %d", res->len, flt->res->len, flt->cnd->tag, flt->set, 0!=flt->mrk, dim); if ( !flt->res->len ) goto done; /* filter by sweep, costing O(flt->res->len + res->len) for res->len << flt->res->len/ld(flt->res->len), binsearch could be cheaper however, sweep is much simpler with varying dimensions and for marking, we'll do a full pass anyway TODO: lotsa other optimizations possible here obviously we could pull several case distinctions out of the loop to trade code size for speed */ for ( ; p < pe; p++ ) { /* advance f, while it is too low */ lulu minrit = p->rit; if ( !dim ) minrit &= RIDMSK; while ( f->rit < minrit ) if ( fe == ++f ) goto done; if ( (RIDMSK&f->rit) > p->rit ) goto nomatch; if ( !dim ) goto match; if ( f->rit > p->rit ) goto nomatch; if ( 1 == dim ) goto match; /* got same rit: rid+tag */ pos = OCCMSK & p->pos; if ( 3 == dim && pos < p->pos - flt->cnd->dst ) pos = p->pos - flt->cnd->dst; while ( f->pos < pos ) { if ( fe == ++f ) goto done; if ( f->rit != p->rit ) goto nomatch; } /* still same rit */ if ( f->pos > (p->pos|0xffff) ) goto nomatch; /* occ mismatch */ /* and same occ */ if ( 2 == dim ) goto match; if ( f->pos > p->pos + flt->cnd->dst ) goto nomatch; /* far out */ if ( OP_DISTMAX == flt->cnd->tag ) goto match; /* the strange OP_DISTEX -- alas */ if ( f->pos == p->pos - flt->cnd->dst ) { /* lower match */ if ( !flt->mrk ) goto match; flt->mrk[f - flt->res->ptr] = flt->set; } if ( !(OCCMSK & (p->pos ^ (pos = p->pos + flt->cnd->dst))) ) { /* no ovfl */ Llp *f1 = f; /* scan for upper match */ while ( pos != f1->pos ) { if ( fe == ++f1 || f1->rit != p->rit || f1->pos > pos ) goto nomatch; } if ( !flt->mrk ) goto match; flt->mrk[f1 - flt->res->ptr] = flt->set; } /* had nomatch or marked marking match: go on */ nomatch: if ( flt->mrk ) continue; if ( flt->set ) continue; goto addit; match: /* filter matched */ if ( !flt->mrk ) { if ( flt->set ) { addit: if ( d < p ) *d = *p; d++; } continue; } LOG_DBG(LOG_DEBUG, "marking"); /* marking */ m = flt->mrk + (f - flt->res->ptr); switch (dim) { case 0: while ( !(RIDMSK & (p->rit ^ f->rit)) ) { MCNT *m++ = flt->set; if ( fe == ++f ) goto done; } break; case 1: while ( p->rit == f->rit ) { MCNT *m++ = flt->set; if ( fe == ++f ) goto done; } break; case 2: while ( p->rit == f->rit && !(OCCMSK & (p->pos^f->pos)) ) { MCNT *m++ = flt->set; if ( fe == ++f ) goto done; } break; case 3: /* DISTMAX only -- DISTEX done above */ if ( OCCMSK & (p->pos ^ (pos = p->pos + flt->cnd->dst)) ) /* overflow */ pos = p->pos | 0xffff; while ( p->rit == f->rit && !(OCCMSK & (p->pos^f->pos)) ) { MCNT *m++ = flt->set; if ( fe == ++f ) goto done; } break; } } /* for p < pe */ done: if ( flt->mrk ) { #ifndef NDEBUG eRr(LOG_VERBOSE, "marked %d", mcnt); #endif return; } if ( !flt->set && p < pe ) { /* negative filter exhausted */ if ( d == p ) return; /* none filtered -- keep full set */ memmove(d, p, (pe-p)*sizeof(*p)); d += pe-p; } res->len = d - res->ptr; } /* rMatch */ /* projection to rit (2 dimensions rid+tag) */ static void rProj2 (Res *res) { Llp *p = res->ptr, *e = p + res->len, *d; lulu last = 0; for (;;) { if ( e == p ) return; if ( last == p->rit ) break; last = p++->rit; } /* now p is 1st redundant */ for (d = p; ++p < e;) if ( last != p->rit ) { last = p->rit; *d++ = *p++; } res->len = d - res->ptr - 1; } /* rProj2 */ static lulu rMerge (Res *res, Res *add, int dim) { lulu brk = 0; Llp *a = res->ptr + res->len-1, *end = res->ptr + ses->s; Llp *b, *set; if ( !res->len ) { if ( 2 > dim ) rProj2(res); memcpy(res, add, 4+add->len*sizeof(Ptr)); return res->len; } if ( 3 != dim ) { /* reduce the add set */ static const int tag[] = {OP_AND, OP_SAMETAG, OP_SAMEREP}; Bop bop; Flt flt; bop.tag = tag[dim]; flt.res = res; flt.cnd = &bop; flt.mrk = 0; flt.set = 0; rMatch(add, &flt); } LOG_DBG(LOG_DEBUG, "merging %d+%d dim %d", res->len, add->len, dim); b = add->ptr + add->len-1; for (set = a + add->len; set > a; set--) { Llp *gt = PGT(a, b) ? a : b; #if 0 LOG_DBG(LOG_DEBUG, "a %d b %d set %d use %c", a+1-res->ptr, b+1-add->ptr, set-res->ptr, b==gt ? 'b' : 'a'); #endif if ( set < end ) *set = *gt; else brk = gt->rit; if ( b == gt ) b--; /* if used last b, should bail out on set == a */ else if (res->ptr == a--) { /* used last a */ /* now the remaining bs are to become first in result */ LOG_DBG(LOG_DEBUG, "copying %d bs to %d slots", b+1-add->ptr, set-res->ptr); memcpy(res->ptr, add->ptr, ((char*)(b+1)) - ((char*)add->ptr)); break; } } LOG_DBG(LOG_DEBUG, "merged %d+%d brk %d untouched %d b %d", res->len, add->len, (unsigned)(brk>>16), set+1-res->ptr, b+1-add->ptr); if ( !brk ) { res->len += add->len; return 0; } /* kill entries on brk rid -- find last valid */ brk &= ~RIDMSK; for ( end = res->ptr+ses->s; end-- > res->ptr && brk < end->rit; ) ; res->len = end + 1 - res->ptr; return brk; } /* rMerge */ typedef struct { /* search context */ QLoop qlp; Res *res; /* array to fill in callback */ Qry *qry; /* query */ Tfl *tfl; /* current tag filter */ int dim; /* required dimension */ Flt flt; /* filter or mark set */ int dsc; /* discard filter */ } Scx; static int scb (Scx *scx) /* qLoop callback: search one term */ { Res tmp; /* merge buffer */ unsigned i; const unsigned char *v = scx->qlp.vals; int typ = scx->qlp.qdx->ptr; Llp *p = tmp.ptr; eRr(LOG_VERBOSE, "scb '%b' %d", scx->qlp.cur.len, scx->qlp.cur.byt, scx->qlp.nvals); /* for tag-sorted (non FULTXT) index, we should pre-skip to tag */ for ( i=scx->qlp.nvals; i--; ) { #ifdef CPU_BIG_ENDIAN enum { RID5, RID4, RID3, RID2, RID1, RID0, TAG1, TAG0 }; enum { POS3, POS2, POS1, POS0 }; #else enum { TAG0, TAG1, RID0, RID1, RID2, RID3, RID4, RID5 }; enum { POS0, POS1, POS2, POS3 }; #endif unsigned char *rit = (unsigned char*)&p->rit; unsigned char *pos = (unsigned char*)&p->pos; p->rit = p->pos = 0; if ( !(QDX_FULTXT & typ) ) /* leading tag */ switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: rit[TAG1] = *v++; case QDX_TAG1: rit[TAG0] = *v++; } switch ( QDX_RIDMSK&typ ) { case QDX_RID6: rit[RID5] = *v++; case QDX_RID5: rit[RID4] = *v++; case QDX_RID4: rit[RID3] = *v++; case QDX_RID3: rit[RID2] = *v++; rit[RID1] = *v++; rit[RID0] = *v++; } if ( QDX_FULTXT & typ ) switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: rit[TAG1] = *v++; case QDX_TAG1: rit[TAG0] = *v++; } switch ( QDX_POSMSK&typ ) { case 4: pos[POS3] = *v++; case 3: pos[POS2] = *v++; case 2: pos[POS1] = *v++; case 1: pos[POS0] = *v++; } /* eRr(LOG_VERBOSE, "rid %d tag %d pos %d", (int)(p->rit >> 16), (int)(0xffff & p->rit), p->pos); */ if ( !scx->tfl ) goto tagok; /* find in tag filter */ if ( 1 == scx->tfl->len ) { if ( (unsigned short)p->rit == *scx->tfl->lst ) goto tagok; } else { unsigned short *t = scx->tfl->lst, *te = t + scx->tfl->len; while ( t < te ) if ( (unsigned short)p->rit == *t++ ) goto tagok; } continue; tagok: if ( p->rit < scx->qry->skp ) continue; if ( scx->qry->brk && p->rit >= scx->qry->brk ) break; p++; } if ( !(tmp.len = p - tmp.ptr) ) goto done; if ( scx->flt.res ) { rMatch(&tmp, &scx->flt); if ( scx->flt.mrk ) goto done; } if ( scx->res->len ) rMerge(scx->res, &tmp, scx->dim); else memcpy(scx->res, &tmp, 4+tmp.len*sizeof(Llp)); done: return 0; } /* scb */ static Res *search (Scx *scx, Exp *exp) /* search one expression */ { Res *a = 0, *b; int dim; if ( scx->flt.res ) eRr(LOG_VERBOSE, "%sing %d %c on %d", scx->flt.mrk?"mark":"filter", exp->tag, scx->flt.set?'+':'-', scx->flt.cnd->tag); else eRr(LOG_VERBOSE, "creating %d", exp->tag); if ( OP_IDENT <= exp->tag ) { /* simple expression */ scx->qlp.flg = 0; if ( OP_IDENT < exp->tag ) { qMkKey(scx->qlp.qdx, &scx->qlp.key, exp->trm.val, exp->trm.len); switch (exp->tag) { case T_EQ: scx->qlp.flg = QEQ; break; case T_PFX: scx->qlp.flg = QPF; break; } } else { scx->qlp.flg = QUPTO; qMkKey(scx->qlp.qdx, &scx->qlp.key, exp->bop.lft->trm.val, exp->bop.lft->trm.len); qMkKey(scx->qlp.qdx, &scx->qlp.to, exp->bop.rgt->trm.val, exp->bop.rgt->trm.len); } if ( !(QDX_FULTXT & scx->qlp.qdx->ptr) ) { if (!scx->tfl || 1 != scx->tfl->len) { eRr(ERR_INVAL, "need tag filter with non-fulltext db"); return 0; } } if ( !scx->flt.mrk ) scx->res = a = rAlloc(); qLoop(&scx->qlp); if ( a ) eRr(LOG_VERBOSE, "got %d", a->len); if ( scx->dsc ) { rFree(scx->flt.res); scx->flt.res = 0; } return a; } switch ( exp->tag ) { case OP_NULL: return 0; case OP_HEAD: return search(scx, exp->bop.rgt); case OP_OR: { int dsc = scx->dsc; scx->dsc = 0; /* delay discarding filter */ a = search(scx, exp->bop.lft); scx->dsc = dsc; b = search(scx, exp->bop.rgt); if ( b ) { if ( a ) rMerge(a, b, scx->dim); rFree(b); } eRr(LOG_VERBOSE, "got %d", a->len); return a; } /* case OR */ case OP_TAGS: { Tfl *tfl = scx->tfl; scx->tfl = &exp->tfl; a = search(scx, exp->tfl.lft); scx->tfl = tfl; return a; } /* case TAGS */ /* case OP_NOT: TODO: special case marking not impl. return 0; */ default: if ( (dim = scx->dim) < opdim[exp->tag] ) scx->dim = opdim[exp->tag]; /* if not filtering and the right hand does not support direct marking, create it and use as left hand filter */ if ( !scx->flt.res && !canmark(exp->bop.rgt) ) { scx->flt.res = b = search(scx, exp->bop.rgt); /* create right hand */ scx->flt.cnd = &exp->bop; scx->flt.mrk = 0; scx->flt.set = OP_NOT != exp->tag; a = search(scx, exp->bop.lft); scx->flt.res = 0; break; } /* Else create the left hand (using any filter) */ if ( (a = search(scx, exp->bop.lft)) ) { Flt sav, use; use.res = a; use.cnd = &exp->bop; use.mrk = RES_MARKS(a); use.set = OP_NOT != exp->tag; memset(use.mrk, !use.set, ses->s); sav = scx->flt; /* evaluate the right hand */ if ( canmark(exp->bop.rgt) ) { /* using direct marking, if possible */ scx->flt = use; search(scx, exp->bop.rgt); /* direct marking on a */ } else { /* create it */ scx->flt.res = 0; /* non filtering */ if ( (b = search(scx, exp->bop.rgt)) ) { rMatch(b, &use); /* and mark */ rFree(b); } } if ( a->len ) { /* reduce the marked set */ Llp *p = a->ptr, *e = p + a->len, *d; char *m = use.mrk; eRr(LOG_VERBOSE, "reducing a->len %d", a->len); while ( *m++ ) if ( e == ++p ) goto done; eRr(LOG_VERBOSE, "1st %d were good", p - a->ptr); /* now *p is 1st unmarked */ for ( m--, d=p; plen = d - a->ptr; eRr(LOG_VERBOSE, "... to a->len %d", a->len); } done: scx->flt = sav; } ; } scx->dim = dim; return a; } /* search */ /* parse query and execute search from 'Q*' msg */ static Qry *Q ( Dbo *self, Msg msg ) { Qry *q = 0, *p; char *exp = msg->val+2; int l = msg->len-2, i = 0; if ( !ses->qry ) ses->qry = mAlloc(ses->q * sizeof(Qry)); if ( !self->qry ) self->qry = &((Prv*)self)->qry; #ifdef REFS_ARE_NOW_FIXED if ( l && '#' == *exp && l-1 == a2il(exp+1,l-1,&i) ) { /* plain #n ref */ p = self->qry; if ( 1 == l ) ((Prv*)self)->qry.cur = 0; /* reset 0 query */ if ( (unsigned)i == p->qid ) return p; for ( ; (q = p->nxt); p=q ) if ( (unsigned)i == q->qid ) { p->nxt = q->nxt; /* unlink */ q->nxt = self->qry; /* put at front */ return self->qry = q; } return 0; } #endif /* create new query */ p=q=ses->qry + ses->q -1; do { /* find slot */ if ( !p->dbo ) { /* unused */ q = p; goto gotslot; } if ( q->use < p->use ) q = p; } while ( ses->qry < --p ); /* unlink the lrused */ { Dbo *dbo = q->dbo; if ( q == dbo->qry ) dbo->qry = 0; /* kill current */ else { for (p = dbo->qry; q != p->nxt; ) if ( !(p = p->nxt) ) { eRr(ERR_IDIOT, "unlinked"); return 0; } p->nxt = q->nxt; /* unlink */ } mFree(q->txt); mFree(q->res); mFree(q->buf); } memset(q, 0, sizeof(*q)); gotslot: /* got clean q */ q->dbo = self; q->qid = self->qid+1; q->buf = mAlloc(l); /* see notes on buffer usage at parse */ if ( 0 > (i = parse(q, (unsigned char*)(q->txt = mDupz(exp, l)))) ) goto cleanup; q->sln = i; q->skp = 0x10000; /* always skip rid 0 entries */ if ( !q->exp->tag ) /* the NULL search: loop all */ q->len = self->db->rdx.mid; else { Scx scx; Res *res; lulu last = 0, *d; Llp *r, *e; memset(&scx, 0, sizeof(scx)); scx.qlp.qcb = (QCb*)scb; scx.qlp.qdx = &self->db->qdx; scx.qry = q; if ( !(res = search(&scx, q->exp)) || !res->len ) goto cleanup; d = q->res = mAlloc(res->len * sizeof(lulu)); r = res->ptr; e = r + res->len; do if ( last != (*d = r->rit >> 16) ) last = *d++; while (++r < e); rFree(res); q->len = d - q->res; } if ( '?' == q->txt[i] ) i = parse(q, (unsigned char*)q->txt+i+1); else { q->exp = 0; MFREE(q->buf); } self->qid++; q->nxt = self->qry; /* link */ self->qry = q; return q; cleanup: mFree(q->txt); mFree(q->res); mFree(q->buf); memset(q, 0, sizeof(*q)); return 0; } /* Q */ /* apply filter expression to record rid. return 1, if record passed, else 0. */ static int filter ( Dbo *self, lulu rid, Exp *e ) { if ( !e ) { eOut(0, "%u", (unsigned)rid); return 1; } if ( !e->tag ) { if ( 0 <= rRead(&env.out->lst, &self->db->rdx, rid, self->p) ) { SINK(env.out); return 1; } return 0; } return 1; } /* filter */ typedef struct { /* search context */ QLoop qlp; int lim; /* limit #terms */ int cnt; /* count for term so far */ int det; /* detailled: restrict to tag, count precise rids */ int rln; /* length of rid: 3..6 */ int tln; /* length of tag to match: 0..2 */ int rof; /* offset of rid in vals: 0 or taglength */ int tof; /* offset of tag in vals: 0 or ridlength */ char rid[6]; /* last rid seen */ char tag[2]; /* tag to filter on */ char key[512]; /* decoded term */ int kln; /* key length */ } Tcx; static int tcb (Tcx *tcx) /* qLoop callback: list terms */ { const unsigned char *v, *e; char *rid; int vsz; if ( !(QSAME & tcx->qlp.flg) ) { /* new term */ if ( tcx->cnt ) eOut(0, "%d\011%.*s", tcx->cnt, tcx->kln, tcx->key); if ( !tcx->lim-- ) return 1; /* break loop */ tcx->kln = qRdKey(tcx->qlp.qdx, tcx->key, sizeof tcx->key, &tcx->qlp.cur); tcx->cnt = 0; } if ( !tcx->det ) { tcx->cnt += tcx->qlp.nvals; return 0; } /* detailled */ rid = tcx->rid; if ( !(QSAME & tcx->qlp.flg) ) memset(rid, 0, tcx->rln); vsz = tcx->qlp.qdx->vsz; v = tcx->qlp.vals; e = v + tcx->qlp.nvals*vsz; for ( ; v < e; v += vsz ) if ( !(tcx->tln && memcmp(v+tcx->tof, tcx->tag, tcx->tln)) && memcmp(v+tcx->rof, rid, tcx->rln) ) { tcx->cnt++; rid = (char*)v+tcx->rof; } if ( tcx->rid != rid ) memcpy(tcx->rid, rid, tcx->rln); return 0; } /* tcb */ /* */ static int T ( Db *db, Fld *msg ) /* terms */ { char *v = msg->val+2, *e = msg->val + msg->len, *p; Tcx tcx; memset(&tcx, 0, sizeof(tcx)); tcx.qlp.qcb = (QCb*)tcb; tcx.qlp.qdx = &db->qdx; tcx.qlp.flg = QPF; if ( !(tcx.lim = ses->s) ) tcx.lim = 1; if ( v < e ) { for (p = v; v < e && TAB != *v; v++) ; qMkKey(tcx.qlp.qdx, &tcx.qlp.key, p, v-p); if ( v++ < e ) { for (p = v; v < e && TAB != *v; v++) ; if ( v == p ) tcx.qlp.flg = QLOOP; /* endless */ else { tcx.qlp.flg = QUPTO; qMkKey(tcx.qlp.qdx, &tcx.qlp.to, p, v-p); } if ( v++ < e ) { /* detailled */ int typ = tcx.qlp.qdx->ptr, tag; if ( QDX_LEAFPV & db->qdx.typ ) { eOut(0, "\011-1\011plain values"); return ERR_INVAL; } a2il(v, e-v, &tag); tcx.det = !0; tcx.rln = QDX_RIDMIN+((QDX_RIDMSK&typ)>>QDX_RIDSH); tcx.tln = (QDX_TAGMSK&typ)>>QDX_TAGSH; if ( !(QDX_FULTXT & typ) ) tcx.rof = tcx.tln; else if ( tag ) tcx.tof = tcx.rln; else tcx.tln = 0; if ( tcx.tln ) { /* get the tag bytes */ Val val; Ptr ptr; ptr.tag = (unsigned short)tag; qMkVal(&val, &ptr, typ); memcpy(tcx.tag, val.byt+tcx.tof, tcx.tln); } } } } eOut(0, "#\0110"); /* plain ok comment */ qLoop(&tcx.qlp); if ( tcx.cnt ) eOut(0, "%d\011%.*s", tcx.cnt, tcx.kln, tcx.key); return 0; } /* T */ enum { IDX_FIELD, IDX_WORD, IDX_SPLIT }; typedef struct { QSet qst; int mod; int mhd; } Idx; /* process control fields for X method if a tag is found, return it's pos */ static char *idxctl (Idx *i, Fld *f) { char *p = f->val, *e = p+f->len, *t = p; for ( ; pmod = IDX_FIELD; break; case 'w': i->mod = IDX_WORD; break; case 's': i->mod = IDX_SPLIT; break; case 'a': i->qst.del = 0; break; case 'd': i->qst.del = 1; break; case 'm': i->mhd = t>p+1 && 'H'==p[1]; continue; case 'p': if ( t == p+1 ) i->qst.pfx.len = 0; else { int len = t-p-1; if ( 255 < len ) len = 255; memcpy(i->qst.pfx.byt, p+1, i->qst.pfx.len=len); } continue; case 'r': a2il(p+1, t-p-1, (int*)&i->qst.ptr.rid); continue; case '+': i->qst.del = 0; if (CT_D == lat1ct[(unsigned char)*++p]) return p; continue; case '-': i->qst.del = 1; if (CT_D == lat1ct[(unsigned char)*++p]) return p; continue; default: if (CT_D == lat1ct[(unsigned char)*p]) return p; continue; } a2il(p+1, t-p-1, (int*)&i->qst.ptr.pos); } return 0; } /* idxctl */ /* write a record. header must have 'W' prestripped. */ static int W ( Db *db, Fld *rec ) { int rid = 0; int pos = -1; unsigned siz = 0; char *e = rec->val + rec->len; /* strip rid@pos */ rec->val += a2il(rec->val, rec->len, &rid); if ( e > rec->val && '@' == *rec->val ) { rec->val++; rec->val += a2il(rec->val, e-rec->val, &pos); } if ( e > rec->val && TAB == *rec->val ) rec->val++; rec->len = e-rec->val; if ( 0 < (rid = rWrite(&db->rdx, rec, rid, pos, siz)) ) ses->rid = (unsigned)rid; return rid; } /* W */ /* */ int OPT_STDCALL oDbo ( Dbo *self, Msg msg ) { eRr(LOG_VERBOSE, "oDBo '%.*s'", msg->len, msg->val); if ( ! msg->len ) goto simplewrite; switch (*msg->val) { case 'Q': { Qry *q; int i = ses->r; unsigned len; if ( 1 == msg->len ) { if ( !(q = self->qry) ) return eRr(ERR_BADF, "query expired"); } else if ( TAB != msg->val[1] ) break; else if ( !(q = Q(self, msg)) ) return ERR_INVAL; /* fetch results */ len = q->res ? q->len : (unsigned)self->db->rdx.mid; q->use = ses->req; eRr(LOG_VERBOSE, "%d of %d results off %d", i, q->len, q->cur); eOut(0, "#\t%d\t%d\t%d", len-q->cur, q->qid, (unsigned)(q->brk>>16)); if ( q->res ) while (q->cur < len && (i -= filter(self, q->res[q->cur++], q->exp))) ; else while (q->cur < len && (i -= filter(self, ++q->cur, q->exp))) ; return 0; } /* Q */ case 'R': if ( 1 == msg->len || TAB == msg->val[1] ) { int rid, ret = 0; EADD(0, "W", 1); if ( 1 == msg->len ) { /* R multi */ Fld *end = REND(msg); unsigned mr = ses->r; while ( ++msg < end && 0 <= (ret = ( a2il(msg->val, msg->len, &rid), rRead(&env.out->lst, &self->db->rdx, rid, self->p))) && --mr /* allowing 4 billion if mr was 0 */ ) SINK(env.out); return ret; } else { /* single or range */ char *p = msg->val+2, *e = p + msg->len -2; int count = 1; p += a2il(p, e-p, &rid); if ( rid > self->db->rdx.mid ) return 0; if ( e > p && TAB == *p ) { p++; a2il(p, e-p, &count); if ( 0 >= count || count > (int)ses->r ) count = (int)ses->r; if ( !count || count > self->db->rdx.mid - rid + 1 ) count = self->db->rdx.mid - rid + 1; } for ( ; 0 <= (ret = rRead(&env.out->lst, &self->db->rdx, rid, self->p)) && --count; rid++ ) { eRr(LOG_INFO, "R %u: %d lst %d", rid, ret, LLEN(&env.out->lst)); SINK(env.out); } return ret; } } break; case 'T': if ( 1 == msg->len || TAB == msg->val[1] ) return T(self->db, msg); break; case 'W': if ( 1 == msg->len ) { /* multi */ int left = RLEN(msg)-1, l, ret; EADD(0, "R", 1); for ( msg++; left; left -= l, msg += l ) { if ( !(l = RLEN(msg)) ) msg->tag = -(l = left); else if ( l < 0 || left < l ) return eRr(ERR_INVAL, "whoopsy! bad embedded rec"); if ( 0 > (ret = W(self->db, msg)) ) return ret; self->rid = (unsigned)ret; eOut(0, "%d", ret); } return 0; } else if ( TAB == msg->val[1] ) { /* single */ msg->val += 2; /* strip W */ msg->len -= 2; goto simplewrite; } /* else it's not a 'W' ... */ break; case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': /* well ... we really should not see such messages ... alas */ simplewrite: { int ret = W(self->db, msg); if ( 0 < ret ) { eOut(0, "R\t%d", ret); self->rid = (unsigned)ret; } return 0; } case 'X': if ( 1 == msg->len || TAB == msg->val[1] ) { char buf[0x2000], *mhd = buf, *val; int mhdlen = sizeof(buf); Fld *end = REND(msg); int ctl = 1, lasttag = -1, cnt = 0, len = 0; Idx idx; memset(&idx, 0, sizeof(idx)); idx.qst.qdx = &self->db->qdx; idx.qst.ptr.rid = self->rid; for (;;) { if ( !ctl ) { idx.qst.ptr.tag = msg->tag; val = msg->val; len = msg->len; } else if ( (val = idxctl(&idx, msg)) ) { char *e = msg->val + msg->len; int tag; val += a2il(val, e-val, &tag); idx.qst.ptr.tag = (unsigned short)tag; if ( val < e && TAB == *val ) val++; len = e - val; } if ( val ) { if ( lasttag != idx.qst.ptr.tag ) { idx.qst.ptr.pos = 0; lasttag = idx.qst.ptr.tag; } if ( idx.mhd ) { char *ev = val+len, *d; if ( len > mhdlen ) { if ( mhd != buf ) mFree(mhd); mhd = mAlloc(len); } for (d = mhd;;val++) { char *s = val; while ( val < ev && '<' != *val ) val++; if ( val > s ) { memcpy(d, s, val - s); d += val - s; } while ( val < ev && '>' != *val && '=' != *val ) val++; if ( val == ev ) break; if ( '>' == *val ) continue; s = ++val; while ( val < ev && '>' != *val ) val++; if ( val > s ) { memcpy(d, s, val - s); d += val - s; } } val = mhd; len = d - mhd; eRr(LOG_DEBUG, "mhd len %d", len); } if ( IDX_SPLIT == idx.mod ) cnt += qSetKeyVals(&idx.qst, val, len); else { if ( !qSetKeyVal(&idx.qst, val, len) ) cnt++; if (IDX_FIELD == idx.mod) idx.qst.ptr.pos = (idx.qst.ptr.pos + 0x10000) & OCCMSK; } } if ( end == ++msg ) break; ctl = !msg->tag; } if ( mhd != buf ) mFree(mhd); eOut(0, "#\t%d", cnt); return 0; } /* case 'X' */ } /* switch */ return oStruct((Struct*)self, msg); } /* oDbo */ extern Dbo *newDbo ( char *name ) { Prv *prv; Db *db = dOpen(name); if ( !db ) return 0; prv = mAlloc(sizeof(Prv)); prv->dbo.snd = (disp*)oDbo; prv->dbo.db = db; prv->dbo.qry = &prv->qry; prv->qry.dbo = &prv->dbo; prv->qry.exp = prv->qry.ebf; /* OP_NULL filter */ return &prv->dbo; } /* newDbo */ mlt/hash.c0100644000076400001440000001652510175233664011607 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: hash.c,v 1.1 2005/01/24 18:00:20 kripke Exp $ the hashtable */ #include "pw.h" /* ************************************************************ private functions */ /* add a bucket */ #define HLNEWF( l, t ) do { \ if ( (l)->fav || lExtend( l, 0, 1 ) ) { \ Fld *__f = (l)->fld - (l)->fld->tag--; \ __f->tag = t; \ __f->val = 0; \ __f->len = 0; \ (l)->fav --; \ } } while(0) /* the cdb hash function ( without h/256 ) */ #define CDB_CALC(i,s) i ^= ( i += i << 5, (int) *s++ ) static OPT_INLINE int hs ( const char *s, register int len ) { register int i = 5381, n = (len + 7 ) / 8; if ( ! len ) return i; switch ( len % 8 ) { case 0: do { CDB_CALC(i,s); case 7: CDB_CALC(i,s); case 6: CDB_CALC(i,s); case 5: CDB_CALC(i,s); case 4: CDB_CALC(i,s); case 3: CDB_CALC(i,s); case 2: CDB_CALC(i,s); case 1: CDB_CALC(i,s); } while ( --n ); } return i; } /* hs */ /* lookup key in a bucket, if found the corresponding entry is assigned to res */ static OPT_INLINE HEntry * gethe ( Fld *fld, int hv, const char *key, int len ) { int i; register const char *p, *q; HEntry *he; if ( fld->len ) { for ( he = (HEntry *) fld->val; he; he=he->nxt ) { if ( hv != he->hv || ( i = len ) != he->len ) continue; for ( p = key, q = he->key, --i; *p++ == *q++; --i ) { if ( !i ) { return he; } } } } return 0; } /* gethe */ /* rebuild the table. this is done if: 1. the load factor is > HLOADMAX, or 2. the fragmentation is > HMAXFRAG mode signals if the number of buckets has to be extended or to be reduced, this is done if less than HSHRINKLOAD of the buckets are used */ static void OPT_STDCALL hRebuild ( HTable *table, int mode ) { int i, nsiz, osiz = HSIZ(table); HEntry *ohe, *nhe; Fld *fld; List *nl = (List * )mAlloc(sizeof(List)); HSTAT(table) #ifndef NDEBUG table->coll = 0; #endif if ( mode ) { nsiz = osiz << HSHIFT; table->mask |= ( table->mask <<= HSHIFT, 1 ); } else if ( HSHRINKLOAD > HCURLOAD(table) && HINITIAL < HSIZ(table) ) { nsiz = (unsigned) osiz >> HSHIFT; table->mask = (unsigned) table->mask >> HSHIFT; } else nsiz = osiz; lInit( nl, "%.*s", table->list->fld->len, table->list->fld->val ); lExtend( nl, 0, nsiz ); for ( i = 0; nsiz > i; ++i ) HLNEWF( nl, i+1 ); for ( i = 1; osiz >= i; ++i ) { if ( ( fld = table->list->fld+i)->len ) { for ( ohe = (HEntry *) fld->val; ohe; ohe = ohe->nxt ) { int j = ohe->hv & table->mask; unsigned n = HESIZ(ohe); fld = nl->fld+j+1; if ( LAVL( nl ) >= n || lExtend( nl, n, -1 ) ) { nhe = (HEntry *) nl->buf; memcpy( nhe, ohe, n ); nhe->nxt = 0; #ifndef NDEBUG nhe->dpt = 1; #endif if ( fld->len ) { nhe->nxt = (HEntry *) fld->val; #ifndef NDEBUG nhe->dpt = nhe->nxt->dpt + 1; ++table->coll; #endif } fld->val = (char *) nhe; fld->len += n; nl->buf += n; } } } } table->dsize = 0; lFini( table->list ); mFree( table->list ); table->list = nl; HSTAT(table) } /* hRebuild */ /* ************************************************************ public functions */ /* get */ HEntry * OPT_STDCALL hGet ( HTable *table, const char *key, int len ) { int hv = hs( key, len); return gethe( (table)->list->fld+1+(hv & table->mask), hv, key, len ); } /* hGet */ /* add */ HEntry * OPT_STDCALL hAdd ( HTable *table, const char *key, int len ) { int hv = hs( key, len); Fld *fld = (table)->list->fld+1+(hv & table->mask); HEntry *he = gethe( fld, hv, key, len ); if ( he ) { #ifndef NDEBUG eRr( LOG_DEBUG, "already there %.*s <--> %.*s", he->len, he->key, len, key ); #endif return he; } /* create new entry */ { unsigned need = sizeof(HEntry) + len - HEDEFKEYLEN; if ( LAVL( table->list ) >= need || lExtend( table->list, need, -1 ) ) { he = (HEntry *) table->list->buf; he->nxt = 0; he->obj = 0; #ifndef NDEBUG he->dpt = 1; #endif he->hv = hv; he->len = len; memcpy( he->key, key, len ); if ( fld->len ) { he->nxt = (HEntry *) fld->val; #ifndef NDEBUG ++table->coll; he->dpt = he->nxt->dpt + 1; #endif } fld->val = (char *) he; fld->len += need; table->list->buf += need; ++table->nume; if ( HMAXLOAD < HCURLOAD(table) ) { hRebuild( table, 1 ); return gethe ( (table)->list->fld+1+(hv & table->mask), hv, key, len ); } return he; } } return 0; } /* hAdd */ HEntry* OPT_STDCALL hSet ( HTable *t, const char *key, int len, Obj *o ) { HEntry *he = hAdd(t, key, len); if ( he ) he->obj = o; return he; } /* hSet */ /* delete entry and check fragmentation */ int OPT_STDCALL hDel ( HTable *table, const char *key, int len ) { int hv = hs( key, len); Fld *fld = (table)->list->fld+1+(hv & table->mask); HEntry *he = gethe( fld, hv, key, len ); if ( he ) { int siz = HESIZ(he); HEntry *prev = (HEntry *) fld->val; if ( prev == he ) { fld->val = (char *)he->nxt; } else { for ( ; prev->nxt != he; prev = prev->nxt ); prev->nxt = he->nxt; } #ifndef NDEBUG if ( fld->val ) { ((HEntry *) fld->val)->dpt--; --table->coll; } #endif fld->len -= siz; --table->nume; table->dsize += siz; if ( HMAXFRAG < HCURFRAG(table) ) hRebuild( table, 0 ); return 1; } return 0; } /* hDel */ /* table initialisation */ HTable * OPT_STDCALL hInit ( HTable *table, const char* header ) { int i; table->list = (List * ) mAlloc( sizeof( *table->list ) ); lInit( table->list, header ); for ( i = 0; HINITIAL > i; ++i ) HLNEWF( table->list, i+1 ); table->mask = HINITIAL-1; table->nume = table->dsize = 0; #ifndef NDEBUG table->coll = 0; #endif return table; } /* hInit */ /* free the list the table itself is not freed */ void OPT_STDCALL hFini ( HTable *table ) { lFini( table->list ); mFree( table->list ); table->mask = table->nume = table->dsize = 0; #ifndef NDEBUG table->coll = 0; #endif } /* hFini */ #ifndef NDEBUG /* hashtable statistics */ void OPT_STDCALL hstat ( HTable *table ) { int j, i, n = HSIZ(table); HEntry *he; for ( i = 1, j = 0; n >= i; ++i ) { he = (HEntry *) (table->list->fld+i)->val; if ( he ) j = he->dpt > j ? he->dpt : j; } eRr( LOG_DEBUG, "hash stats:\n\tnum buckets: %u\n\tnum entries: %u\n\tload: %u\n\tlist buf size: %u\n\tdel size: %u\n\tfrag: %u\n\tcollisions: %u\n\tcollision rate: %u\n\tmaxdepth: %u", HSIZ(table), table->nume, HCURLOAD(table), LSIZ(table->list), table->dsize, HCURFRAG(table), table->coll, (table->coll*100)/( table->nume ? table->nume : 1 ), j ); } /* hstat */ #endif mlt/iif.c0100644000076400001440000002656410175233710011427 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: iif.c,v 1.1 2005/01/24 18:00:40 kripke Exp $ malete IIF(ISO2709/Z39.2)/MARC conversion less general than IIF, but less restrictive than MARC */ #include "tool.h" /* implemented structure of IIF files: RT, FT and delimiter as defined by Z39.2-1979 other chars as used by WinIsis (esp. delimiter ^) are NOT supported the bad habit of WinIsis to insert \r\n every 80 chars is NOT supported Z39.2 specifies that tag 1 is for a control number, every record must have exactly one tag 2 is for subrecord purposes tags 3-9,a-z are other control fields control field tags 001-00z must be first and sorted in dict and data, so base of data contains always the "control number" (not nec. a number) control fields contain neither indicators nor subfields/delimiters, but FT MARC specifies that tag 5 is date and time of latest transaction there are 2 indicators in [ a-z0-9] unused indicators should be blank; printed as # in specs other unused but fixed coded data should use a '|' */ enum { RT = 29, /* ^] record terminator; printed \ in MARC specs */ FT = 30, /* ^^ field terminator; printed ^ or @ in MARC specs */ DL = 31 /* ^_ subfield delimiter; printed $ in MARC specs */ }; typedef struct { /* 24 byte Leader; mostly ASCII digits */ char len[5]; /* record length incl. RT */ char sts; /* status (new, updated); "ASCII graphic" */ char typ; /* type of record; "ASCII graphic" */ char id0[3]; /* impl.def., used by some MARCs */ char ind; /* indicator count; 2 in MARC */ char idl; /* identifier length incl delimiter; 2 in MARC */ char off[5]; /* base address of data = 24 + dict size */ char id1[3]; /* impl.def., used by some MARCs */ /* entry map; lof+scp > 0 */ char lof; /* # bytes for length of field; 4 in MARC */ char scp; /* # bytes for starting character pos; 5 in MARC */ char idp; /* # bytes for impl.def. portion; 0 in MARC */ char res; /* reserved; 0 in MARC */ } Leader; typedef struct { /* dictionary entries after leader, MARC layout */ char tag[3]; /* IIF allows alpha; MARC demands digits */ char lof[4]; /* [lof] length of field incl indicators and FT */ /* if lof is 0, next entry has addtl. lof for same field */ char scp[5]; /* [scp]; starting char pos rel to off */ /* char idp[idp]; impl.def. portion */ } Entry; /* entries followed by an FT */ typedef struct { Fld file; /* points into env->opt */ int rid; /* use control number as MFN */ int nomarc; /* do not assume MARC */ int write; /* write iif */ char ind[2]; char sub; } IifOpt; static int iifOpt (IifOpt *opt, Fld *args) { Fld o; memset(opt, 0, sizeof(*opt)); for ( o.val = 0; vGet(&o, args, "FNPR"); ) switch (o.tag) { case 'F': opt->file = o; continue; case 'N': opt->nomarc = !0; continue; case 'P': opt->ind[0] = opt->ind[1] = ' '; opt->sub = '0'; switch (o.len) { default: opt->sub = o.val[2]; case 2: opt->ind[1] = o.val[1]; case 1: opt->ind[0] = o.val[0]; case 0: ; } continue; case 'R': opt->rid = !0; continue; } return 0; } /* iifOpt */ int iifuse () { return eRr(ERR_INVAL, "[iifimp|iifexp]:\n" "-F: path/filename of iif (default = malete db.iif)\n" #ifndef WIN32 "\t-F- for stdin/out\n" #endif "-Nomarc: do not assume MARC structure 22/450# (imp)\n" "-P[]: prepend indicators ii and, where needed, subfield c (exp)\n" "-Rid: use control number as rid (imp)\n" "use tools like yaz-marcdump for character set recoding\n" ); } /* iifuse */ static int iifOpen (file *iif, IifOpt *opt, Db *db) { int filmode = opt->write ? FIL_RDWR|FIL_CREAT|FIL_TRUNC : FIL_RD; int ret; char *path; if ( opt->file.len ) { #ifndef WIN32 if ( 1 == opt->file.len && '-' == *opt->file.val ) { *iif = opt->write ? 1 : 0; return 0; } #endif path = VDUPZ(&opt->file); } else { int l = strlen(db->pat); path = mAlloc(l+5); memcpy(path, db->pat, l); memcpy(path+l, ".iif", 5); } ret = fOpen(iif, path, filmode) ? eRr(LOG_ERROR, "no access to IIF file '%s'", path) : 0; mFree(path); return ret; } /* iifOpen */ /* tags may contain alpha; uppercase or lowercase ("but not both") UNIMARC and others specify plain numeric tags MARC specs first two to be digits; we map: 3 digits to 0-999 (000 actually forbidden) 2 digits: 3rd 'a' to 1000-1099, 'b' to 1100-1199 up to 3599 1 digit: 1st + 360*2ndbase36 + 10*3rdbase36 up to 12959 others are base 36 (>=12960 = 'a00') < 36^3 = 46656 */ static int rdtag (char *t) { int tag; switch ( a2il(t, 3, &tag) ) { case 3: return tag; case 2: return tag + 100*b36val[(unsigned char)t[2]]; case 1: return tag + 360*b36val[(unsigned char)t[1]] + 10*b36val[(unsigned char)t[2]]; } return 1296*b36val[(unsigned char)t[0]] + 36*b36val[(unsigned char)t[1]] + b36val[(unsigned char)t[2]]; } static OPT_INLINE void pr (char *t, unsigned i, unsigned l) { for ( ; l--; i/=10) t[l] = b36dig[i%10]; } static void prtag (char *t, unsigned tag) { if (1000 > tag) pr(t, tag, 3); else if (3600 > tag) { pr(t, tag%100, 2); t[2] = b36dig[tag/100]; } else if (12960 > tag) { t[0] = b36dig[tag%10]; tag /= 10; t[2] = b36dig[tag%36]; tag /= 36; t[1] = b36dig[tag]; } else { t[2] = b36dig[tag%36]; tag /= 36; t[1] = b36dig[tag%36]; tag /= 36; t[0] = b36dig[tag%36]; } } int iifimp (Db *db, Fld *args) { char buf[99999]; IifOpt opt; file iif; Leader ldr; List rec; int ret = 0; if ( iifOpt(&opt, args) ) return iifuse(); if ( iifOpen(&iif, &opt, db) ) return eRr(ERR_INVAL, "bad IIF base"); #define CLEANUP( args ) do { ret = eRr args; goto cleanup; } while (0) lInit(&rec, ""); for (;;) { char head[8]; char *ent = buf, *base; int len, off, eln, dln, lof = 0, scp = 0; int got = fRead(&iif, &ldr, sizeof(ldr)); int rid = 0; if ( !got ) break; if ( (int)sizeof(ldr) != got ) CLEANUP((ERR_TRASH, "reading leader got %d", got)); LOG_DBG(LOG_TRACE, "leader '%.*s'", 24, &ldr); len = a2i(ldr.len, 5) - 24; off = a2i(ldr.off, 5); if ( opt.nomarc ) { ldr.ind -= '0'; ldr.idl -= '0'; ldr.lof -= '0'; ldr.scp -= '0'; ldr.idp -= '0'; } else { ldr.ind = 2; ldr.idl = 2; ldr.lof = 4; ldr.scp = 5; ldr.idp = 0; } eln = 3+ldr.lof+ldr.scp+ldr.idp; /* entry length */ dln = (off-25)/eln; /* dict len: off - leader - FT */ eRr(LOG_VERBOSE, "len %d sts %c typ %c ind %d idl %d off %d map 3+%d+%d+%d=%d %d fields", len, ldr.sts, ldr.typ, ldr.ind, ldr.idl, off, ldr.lof, ldr.scp, ldr.idp, eln, dln); if ( off != 25+eln*dln ) eRr(ERR_TRASH, "bad off %d != 25+%d*%d", off, eln, dln); got = fRead(&iif, buf, len); if ( len != got ) CLEANUP((ERR_TRASH, "reading body %d got %d", len, got)); base = buf+off-24; if ( FT != base[-1] ) eRr(ERR_TRASH, "no FT after dict"); if ( RT != buf[len-1] ) eRr(ERR_TRASH, "no RT"); memcpy(head, &ldr.sts, 5); /* sts,typ,id0 */ memcpy(head+5, ldr.id1, 3); lClr(&rec); LADD(&rec, -1, head, 8); for ( ; dln--; ent+=eln ) { int tag = rdtag(ent); char *v, *e, *p; if ( ldr.scp ) scp = a2i(ent+3+ldr.lof, ldr.scp); else scp += lof; v = base+scp; if ( !ldr.lof ) { if ( !(e = memchr(v, FT, len-off+24)) ) CLEANUP((ERR_TRASH, "no FT after field")); lof = e-v+1; } else { if ( !(lof = a2i(ent+3, ldr.lof)) ) { /* long field */ int max=1, dig=ldr.lof, add; while (dig--) max *= 10; lof = max; do { ent += eln; if ( !dln-- ) CLEANUP((ERR_TRASH, "eod on long field")); if ( tag != a2i(ent, 3) ) eRr(ERR_TRASH, "bad tag"); add = a2i(ent+3, ldr.lof); lof += add ? add : max; } while ( !add ); } if ( FT != *(e = v+lof-1) ) eRr(ERR_TRASH, "no FT after field"); } for ( p=v; prdx.mid; i++) { /* loop records */ char head[8], rid[12]; char *p; int l, n, o, ll, ridl; Entry *e; Fld *f; if ( 0 >= dRead(lClr(&rec), db, i) ) /* no rec */ continue; ll = LLEN(&rec)-1; /* fill head */ for (p = rec.fld->val, l = rec.fld->len; l-- && TAB != *p++; ) ; if ( 8 <= l ) memcpy(head, p, 8); else { memset(head, ' ', 8); if ( 0 < l ) memcpy(head, p, l); } memcpy(&ldr.sts, head, 5); memcpy(ldr.id1, head+5, 3); e = (Entry*)(buf + 24); /* after leader */ l = 26; /* leader + dict FT + RT */ o = 0; /* offset to base */ n = 0; /* # fields used */ /* rid -> control number */ if ( ll && 1 == rec.fld[1].tag ) ridl = 0; else { ridl = i2a(rid, i); prtag(e->tag, 1); pr(e->lof, o = ridl+1, 4); pr(e->scp, 0, 5); l += o + 12; e++; } for ( f = rec.fld+1; l < 99900 && n < ll; n++,e++,f++ ) { int fln = 1 /* FT */, vln = f->len; l += 12; /* Entry */ if ( f->tag > 9 ) { /* non control */ if ( opt.ind[0] ) fln += 2; if ( opt.sub && vln && TAB != *f->val ) fln += 2; } if ( vln > 9994 ) vln = 9994; /* max field len */ if ( vln > 99994-l ) vln = 99994-l; /* max rec len */ l += fln += vln; prtag(e->tag, f->tag); pr(e->lof, fln, 4); pr(e->scp, o, 5); o += fln; } p = (char*)e; *p++ = FT; pr(ldr.len, l, 5); pr(ldr.off, p - buf, 5); memcpy(buf, &ldr, 24); /* now add the field data */ if ( ridl ) { memcpy(p, rid, ridl); p += ridl; *p++ = FT; } for ( f = rec.fld+1; n--; f++ ) { int vln = f->len; char *v = f->val; if ( f->tag > 9 ) { /* non control */ if ( opt.ind[0] ) { *p++ = opt.ind[0]; *p++ = opt.ind[1]; } if ( opt.sub && vln && TAB != *f->val ) { *p++ = DL; *p++ = opt.sub; } } if ( vln > 9994 ) vln = 9994; if ( vln > 99994-(p-buf) ) vln = 99994-(p-buf); while ( vln-- ) if ( TAB == (*p++ = *v++) ) p[-1] = DL; *p++ = FT; } *p++ = RT; if ( l != p-buf ) { eRr(ERR_IDIOT, "miscalculated len %d got %d rec %d", l, p-buf, i); break; } fWrite(&iif, buf, l); } fClose(&iif); return ret; } /* iifexp */ mlt/malete.c0100644000076400001440000001174610233236727012131 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: malete.c,v 1.2 2005/04/25 15:49:25 kripke Exp $ malete main */ #include "tool.h" static int usage () { eRr( 0, "malete "TOOL_VERSION" core "CORE_VERSION" pw "PW_VERSION #ifndef BUILD_NOCDS " cds" #endif #ifndef BUILD_NOIIF " iif" #endif ); return 0; } static int qCb ( QLoop *self ) { Ptr p; unsigned i, vsz = self->qdx->vsz; const unsigned char *v = self->vals; char buf[512]; int len = qRdKey(self->qdx, buf, sizeof buf, &self->cur); for ( i=self->nvals; i--; v += vsz) { qRdVal(&p, v, self->qdx->ptr); eOut( 0, "%.*s\t%u\t%d\t%u\t%u", len, buf, p.rid, p.tag, p.pos>>16, p.pos&0xffff); } return 0; } /* qCb */ /* ************************************************************ */ int main ( int argc, const char **argv ) { List arg; Ses s; List rec; Fld opt; Db *db[2] = {0,0}; char *p; int i; lInit(&rec, "test"); CINIT(&arg); sInit(ses = &s); if ( LOG_VERBOSE >= env.log ) SCPY(env.err, env.opt); for ( i=1; i<3 && ifld+i) ) { if ( !(db[i-1] = dOpen(p = VDUPZ( &opt ))) ) return 1; mFree( p ); } } if ( !VPRI(&opt, env.opt->fld) ) return usage(); switch ( opt.val[0] ) { case 'c': #ifndef BUILD_NOCDS if ( VEQZ(&opt, "cdsimp") ) { if ( !db[0] ) return cdsuse(); cdsimp(db[0], env.opt->fld); } else if ( VEQZ(&opt, "cdsexp") ) { if ( !db[0] ) return cdsuse(); cdsexp(db[0], env.opt->fld); } #endif break; case 'd': if ( VEQZ(&opt, "dbinfo") ) { if ( !db[0] ) goto dbs1; eRr(0, "db %s path %s has %d records", db[0]->nam, db[0]->pat, db[0]->rdx.mid); i = db[0]->rdx.typ; eRr(0, "mrx type 0x%02x: %d pos + %d len + %d fields = %d bytes", i, 4+(i>>4), 3+(i>>2&3), i&3, db[0]->rdx.ptl); i = db[0]->qdx.typ; eRr(0, "mqd type 0x%02x: blocksize %d, %d byte %s values, %d byte %scompressed keys", i, 1<<((i&QDX_SIZMSK)+QDX_LEAFSH), db[0]->qdx.vsz, QDX_LEAFPV&i ? "plain" : "pointer", db[0]->qdx.ksz, QDX_COMPRS&i ? "" : "un"); i = db[0]->qdx.ptr; if ( !(QDX_LEAFPV&db[0]->qdx.typ) ) eRr(0, "mqd pointers 0x%02x: %d record id + %d tag + %d pos bytes %s first", i, QDX_RIDMIN+((QDX_RIDMSK&i)>>QDX_RIDSH), (QDX_TAGMSK&i)>>QDX_TAGSH, QDX_POSMSK&i, QDX_FULTXT&i ? "tag" : "rid"); if ( db[0]->opt ) SCPYR(env.out, db[0]->opt); } break; case 'h': return usage(); case 'i': #ifndef BUILD_NOIIF if ( VEQZ(&opt, "iifimp") ) { if ( !db[0] ) return iifuse(); iifimp(db[0], env.opt->fld); } else if ( VEQZ(&opt, "iifexp") ) { if ( !db[0] ) return iifuse(); iifexp(db[0], env.opt->fld); } #endif break; case 'q': if ( VEQZ(&opt, "qdump") ) { QLoop ql; if ( !db[0] ) goto dbs1; env.out->snk = fSinkl; eOut(0, ""); /* eats one */ memset( &ql, 0, sizeof(ql) ); ql.qcb = qCb; ql.qdx = &db[0]->qdx; qLoop( &ql ); } else if ( VEQZ(&opt, "qload") ) { if ( !db[0] ) goto dbs1; qLoadf(&db[0]->qdx, &env.in); } else if ( VEQZ(&opt, "qchk") ) { QLoop ql; if ( !db[0] ) goto dbs1; memset( &ql, 0, sizeof(ql) ); ql.qcb = 0; ql.qdx = &db[0]->qdx; qLoop( &ql ); } else if ( VEQZ(&opt, "qset") ) { Key key; FIL_DEFBUF(env.in); if ( !db[0] ) goto dbs1; memset( &key, 0, sizeof(key) ); key.val.len = db[0]->qdx.vsz; while ( fGets(&fb) && fb.l ) { if ( qMkKeyVal(&db[0]->qdx, &key, fb.p, fb.l) ) { eRr(LOG_VERBOSE, "bad key at line %d offset %d", fb.n, fb.o); continue; } if ( qSet(&db[0]->qdx, &key) ) break; } eRr(LOG_INFO, "did %d lines %d bytes", fb.n, fb.o); } break; case 'r': if ( VEQZ(&opt, "rdump") ) { if ( !db[0] ) goto dbs1; for (i=1; i<=db[0]->rdx.mid; i++) { dRead(&env.out->lst, db[0], i); SEOR(env.out); } } else if ( VEQZ(&opt, "rload") ) { FIL_DEFBUF(env.in); while ( fGetr(&env.out->lst, &fb) ) { if ( db[0] ) dWrite(db[0], env.out->lst.fld, 0); SEOR(env.out); } } break; case 's': if ( VEQZ(&opt, "server") ) return server(); } return 0; dbs1: eRr(ERR_INVAL, "command %.*s needs a db", opt.len, opt.val); return 1; } /* main */ mlt/pw.c0100644000076400001440000001220610175233760011277 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: pw.c,v 1.1 2005/01/24 18:01:20 kripke Exp $ Malete patchwork functions */ #include "pw.h" Ses *ses; /* ************************************************************ object */ int OPT_STDCALL mPipe ( Obj *self, Msg msg ) { #if 1 (void)self; (void)msg; return ERR_INVAL; #else int rv = ERR_INVAL, eflg = env.flg; const char *p, *q, *e; Sink l1, l2, *pin, *pout, *eout = env.out; lInit( pout = &l1, 0 ); lInit( pin = &l2, 0 ); lCpy( pin, msg, 0 ); LADD( pout, -1, msg->val, msg->len ); env.out = pout; env.flg |= ENV_BUFFER; for ( p = msg->val, q = 0, e = msg->val + msg->len; e > p; ) { if ( ( TAB == *p++ && ( '|' == *p || msg->val == p-2 ) ) || e == p ) { if ( q ) { char *s = pin->fld->val; pin->fld->val += q - msg->val; pin->fld->len = ( p - q ); if ( e > p ) --pin->fld->len; if ( ( rv = self->snd( self, pin->fld ) ) ) goto cleanup; pin->fld->val = s; pin->fld->len = msg->len; lReset( pin ); env.out = pin; pin = pout; pout = env.out; } q = '|' == *p ? p+1 : p; } } LCPY( eout, pin ); cleanup: env.out = eout; env.flg = eflg; lClr( pin ); lClr( pout ); return rv; #endif } /* mPipe */ int OPT_STDCALL mComp ( Obj *self, Msg msg ) { const char *e = msg->val + msg->len; char *q, *p; int rv = ERR_INVAL; for ( q = 0, p = msg->val, e = msg->val + msg->len; e > p; ) { if ( ( TAB == *p++ && ( ';' == *p || msg->val == p-2 ) ) || e == p ) { if ( q ) { msg->val = q; msg->len = (p - q); if ( e > p ) --msg->len; if ( 0 > (rv = self->snd(self, msg)) ) return rv; } q = ';' == *p ? p+1 : p; } } return rv; } /* mComp */ int OPT_STDCALL mComm ( Obj *self, Msg msg ) { EADDF(msg); if ( -1 <= msg->tag ) return 0; if ( msg[1].tag > msg->tag+1 ) /* not properly embedded */ msg[1].tag = msg->tag+1; return self->snd(self, msg+1); } /* mComm */ int OPT_STDCALL oObj ( Obj *self, Msg msg ) { eRr(LOG_VERBOSE, "oObj '%.*s'", msg->len, msg->val); if ( msg->len ) switch (*msg->val) { case '|': return mPipe(self, msg); case ';': return mComp(self, msg); case '#': return mComm(self, msg); case '=': return 0; case '.': return dispatch(msg); } else return eOut(0, "#\t0"); return eRr(ERR_INVAL, "unrecognized message '%.*s'", msg->len, msg->val); } /* oObj */ /* ************************************************************ struct */ int OPT_STDCALL oStruct ( Struct *self, Msg msg ) { HEntry *he; int nlen; char *p = msg->val, *e = p + msg->len; if ( !msg->len || CT_A != lat1ct[(unsigned char)*p] ) return oObj((Obj*)self, msg); eRr(LOG_VERBOSE, "oStruct '%.*s'", msg->len, msg->val); /* must address child */ for ( ;p < e; p++ ) switch (*p) { case TAB: break; case '.': goto dot; } return eRr(ERR_INVAL, "bad struct msg '%.*s'", p-msg->val, msg->val); dot: nlen = p++ - msg->val; if ( !self->reg || !(he = hGet(self->reg, msg->val, nlen)) ) { Obj *cld; if ( !self->fac || !(cld = self->fac(self, msg->val, nlen)) ) return ERR_INVAL; if ( !self->reg ) { self->reg = (HTable *) mAlloc( sizeof(HTable) ); hInit(self->reg, "reg"); } he = hSet(self->reg, msg->val, nlen, cld); } if ( he && he->obj ) { msg->len -= nlen + 1; msg->val = p; return ((Obj *) he->obj)->snd( (Obj *) he->obj, msg ); } return eRr(ERR_INVAL, "no such child '%.*s'", nlen, msg->val); } /* oStruct */ int OPT_STDCALL oSes ( Ses *self, Msg msg ) { self->req++; if ( msg->len && '.' == *msg->val ) { /* rooted */ msg->val++; msg->len--; } else if (self->at) { /* prepend default */ /* TODO */ } return oStruct((Struct*)self, msg); } /* oSes */ Obj* OPT_STDCALL facMain ( Ses *self, const char *key, int len ) { char *name = mDupz(key, len); Obj *ret; eRr(LOG_VERBOSE, "facMain '%.*s'", len, key); (void)self; /* could be used for authorization etc. */ /* we should scan env.opt for defined objects like remote servers however, db is the default anyway */ ret = (Obj*)newDbo(name); mFree(name); return ret; } /* facMain */ void sInit (Ses *self) { memset(self, 0, sizeof(Ses)); self->snd = (disp*)oSes; self->fac = (factory*)facMain; self->out = env.out; self->err = env.err; /* TODO: check env opts for option settings */ self->r = 20; self->q = 20; self->s = 10000; } /* sInit */ mlt/pw.h0100644000076400001440000002126410207134340011276 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ #ifndef PW_H /* $Id: pw.h,v 1.2 2005/02/23 17:09:52 kripke Exp $ interface of the Malete patchwork */ #define PW_VERSION "1.0" #include "core.h" typedef struct Obj Obj; /* ****************************************************************** */ /* */ /* HASH TABLE */ /* */ /* ****************************************************************** */ #define HEDEFKEYLEN 1 typedef struct HEntry HEntry; /* hmmm ... maybe this should become part of Obj */ struct HEntry { HEntry *nxt; Obj *obj; int hv; /* hash value of this entry */ #ifndef NDEBUG int dpt; #endif int len; char key[HEDEFKEYLEN]; /* not 0 terminated */ }; /* hmmm ... maybe this should become part of Struct */ typedef struct HTable { List *list; int mask; int nume; /* # of entries */ int dsize; /* # bytes of unlinked entries */ #ifndef NDEBUG int coll; /* # of collisions */ #endif } HTable; #define HSIZ(t) (-((t)->list->fld->tag)-1) /* number of buckets */ #define HSHIFT 1 /* rebuild factor: HSIZ(t) << HSHIFT */ #define HESIZ(he) (sizeof(HEntry)+(he)->len-HEDEFKEYLEN) /* entry size */ #define HINITIAL 32 /* initial # of buckets */ #define HMAXLOAD 75 /* max load factor */ #define HCURLOAD(t) ( (100*(t)->nume) / HSIZ(t) ) /* current load */ #define HSHRINKLOAD 25 /* shrink, if HCURLOAD(t) < HSHRINKLOAD */ #define HMAXFRAG 50 /* max fragmentation factor */ #define HCURFRAG(t) ( (100*((t)->dsize)) / LSIZ((t)->list) ) /* current frag */ extern HEntry* OPT_STDCALL hGet ( HTable *table, const char *key, int len ); extern HEntry* OPT_STDCALL hAdd ( HTable *table, const char *key, int len ); extern HEntry* OPT_STDCALL hSet ( HTable *t, const char *key, int len, Obj *o ); extern int OPT_STDCALL hDel ( HTable *table, const char *key, int len ); extern HTable* OPT_STDCALL hInit ( HTable *table, const char* header ); extern void OPT_STDCALL hFini ( HTable *table ); /* free al resources held by table */ #define HFREE( table ) do { \ hFini( (table) ); \ mFree( table ); \ } while (0) #ifndef NDEBUG /* debug statistics */ extern void OPT_STDCALL hstat ( HTable *table ); # define HSTAT(t) hstat(t); #else # define HSTAT(t) #endif /* ****************************************************************** */ /* */ /* OBJECTS AND MESSAGES */ /* */ /* ****************************************************************** */ typedef Fld * Msg; typedef struct Qry Qry; /* query implemented by dbo */ /* dispatch message "msg", starting at object obj. ----------------------------------- Syntax and semantics of messages A message is basically an array of fields - usually of length 1. The leader of this array containes the messagename, denoting a method, and possibly options. The mesagename has to be given in the leader's primary value, the options in following subfields. Data to be processed is contained in the following fields - if any. The messagename may contain subobject specifiers, which have to be separated by a '.'. Besides this the evaluation of the given leader is solely in the responsibility of the method invoked. The message evaluation may change the given message. The significant length of a messagename may be limited to four bytes. A leader should not be longer than DEFBLKLEN bytes, though some messages may allow longer sizes. Some messagenames may use special characters denoting a special treatment of the given leader - thus such characters should not be used for ordinary messagenames. The result of the evaluation, if any, is always written to env.out. */ typedef int OPT_STDCALL disp ( Obj *self, Msg msg ); /* the general form of an object send it a message using that->snd(msg) In general, snd must return <0 on error, >=0 else */ struct Obj { disp *snd; /* method implementation */ }; /** base implementation for Obj handling pipes and such */ extern int OPT_STDCALL oObj ( Obj *self, Msg msg ); typedef struct Struct Struct; typedef Obj* OPT_STDCALL factory ( Struct *self, const char *key, int len ); /* an object with variable number of childs */ struct Struct { disp *snd; /* method implementation */ HTable *reg; /* child registry */ factory *fac; /* child factory */ }; /** base implementation for Struct handling child dispatching */ extern int OPT_STDCALL oStruct ( Struct *self, Msg msg ); /** session -- the root object */ typedef struct Ses { disp *snd; HTable *reg; /* main registry */ factory *fac; /* main factory */ Sink *out; /* main env.out for this session */ Sink *err; /* env.err for this session */ unsigned rid; /* last rid written */ Qry *qry; /* array of q queries */ lolo req; /* request counter */ /* options */ char *at; /* default target */ unsigned r; /* max # of records per read, 0 = unlimited */ unsigned q; /* max # of open query cursors != 0 */ unsigned s; /* max # of results in search set, 0 = unlimited */ } Ses; extern int OPT_STDCALL oSes ( Ses *self, Msg msg ); /* default main factory (i.e. for the session). scans the environment options for defined objects */ extern Obj* OPT_STDCALL facMain ( Ses *self, const char *key, int len ); /* init session */ extern void sInit (Ses *s); extern Ses *ses; /* current session */ /* main dispatcher */ #define dispatch( msg ) ses->snd((Obj*)ses, msg) /* object linked to a session on some remote server */ typedef struct Srv { disp *snd; /* method implementation */ file *con; /* connection */ } Srv; /** interface a stream */ extern int OPT_STDCALL oSrv ( Srv *self, Msg msg ); /* database object NOTE that these are childs of session, so all status is per session. */ typedef struct Dbo { disp *snd; HTable *reg; /* views etc */ factory *fac; /* child factory */ Db *db; /* shared by the Dbos */ Qry *qry; /* chain of queries */ unsigned qid; /* number of last query, 0 = none */ unsigned rid; /* number of last written rec, 0 = none */ /* options */ unsigned p; /* position limit for snapshot read, 0 = none */ } Dbo; /** handle db access */ extern int OPT_STDCALL oDbo ( Dbo *self, Msg msg ); extern Dbo *newDbo ( char *name ); /* constructor */ extern void delDbo ( Dbo *self ); /* destructor */ /* ****************************************************************** */ /* */ /* HANDLERS FOR VARIOUS STANDARD MESSAGES */ /* */ /* ****************************************************************** */ /* Process a pipe message '|'. The leader is parsed for the subfields '|'. Each of which should contain a valid messagename, options may be given in following subfields terminated by the next '|' subfield or the end of the leader. The output of the first method is used as data input for the next and so on. The first submessage may start directly after the primary field w/o the '|' identifier. */ extern int OPT_STDCALL mPipe ( Obj *self, Msg msg ); /* Process a compound message ';'. The leader is parsed for the subfields ';'. Each of which should contain a valid messagename, options may be given in following subfields terminated by the next ';' subfield or the end of the leader. Each message is processed separately, given data is used for all messages. The first submessage may start directly after the primary field w/o the ';' identifier. */ extern int OPT_STDCALL mComp ( Obj *self, Msg msg ); /* Process a comment message '#'. */ extern int OPT_STDCALL mComm ( Obj *self, Msg msg ); #define PW_H #endif /* PW_H */ mlt/qdx.c0100644000076400001440000021477210233236727011462 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: qdx.c,v 1.2 2005/04/25 15:49:25 kripke Exp $ the btree. */ #include #include #include "core.h" /* ************************************************************ private types */ /** Dictionary layout for forks (inner nodes of tree): */ typedef struct Dict { /* dictionary of block entries */ unsigned short pos; /* position of entry, native byte order */ unsigned char vln; /* number of values */ unsigned char kln; /* length of key */ } Dict; /* For leaves, the max block size is 8K, which need 13 bit for pos and 11 bit for vln (max 8K/min vsz 4 values). The 1st 3 bytes are: 1. lower 8 bit of pos 2. higher 5 bit of pos | higher 3 bit of vlen << 5 3. lower 8 bit of vln */ #ifdef CPU_BIG_ENDIAN #define LEAFPOS( d ) (*((unsigned char*)(d)) \ | (((unsigned char*)(d))[1]&0x1f)<<8) #define LEAFVLN( d ) ((d)->vln \ | (((unsigned char*)(d))[1]&0xe0)<<3) #define LEAFPOSADD( d, a ) do { \ unsigned char *__u = (unsigned char*)(d); \ unsigned short __p = (a) + __u[0] + (__u[1]<<8); \ __u[0] = (unsigned char)__p; \ __u[1] = (unsigned char)((__u[1]&0xe0)|((__p>>8)&0x1f)); \ } while (0) #define LEAFVLNADD( d, a ) do { \ unsigned char *__u = (unsigned char*)(d); \ unsigned short __v = (a) + __u[2] + ((__u[1]&0xe0)<<3); \ __u[2] = (unsigned char)__v; \ __u[1] = (unsigned char)((__u[1]&0x1f)|((__v>>3)&0xe0)); \ } while (0) #define LEAFSETPV( d, p, v ) do { \ unsigned char *__u = (unsigned char*)(d); \ __u[0] = (unsigned char)(p); \ __u[1] = (unsigned char)((((p)>>8)&0x1f)|(((v)>>3)&0xe0)); \ __u[2] = (unsigned char)(v); \ } while (0) #else #define LEAFPOS( d ) (0x1fff & (d)->pos) #define LEAFVLN( d ) ((d)->vln | (~0x1fff & (d)->pos)>>5) #define LEAFPOSADD( d, a ) do { \ (d)->pos = (~0x1fff & (d)->pos) | (0x1fff & ((d)->pos + (a))); \ } while (0) #define LEAFVLNADD( d, a ) do { \ unsigned _vln = LEAFVLN( d ) + (a); \ (d)->pos = (0x1fff & (d)->pos) \ | (~0x1fff & ((_vln)<<5)); \ (d)->vln = (unsigned char)(_vln); \ } while (0) #define LEAFSETPV( d, p, v ) do { \ (d)->pos = (0x1fff & (p)) | (~0x1fff & ((v)<<5)); \ (d)->vln = (unsigned char)(v); \ } while (0) #endif #define LEAFSETPVK( d, p, v, k ) do { \ LEAFSETPV( d, p, v ); \ (d)->kln = (unsigned char)(k); \ } while (0) /* the first 8 bytes are actually redundant, however are here for convenience and as a check */ typedef struct Block { unsigned num; /* number of this block, for check and convenience */ unsigned char typ; /* block type & endianess */ unsigned char ksz; /* max key length */ unsigned char ptr; /* ptr type; for forks+plain leaves: value size */ unsigned char lev; /* level over bottom, 0 for leaves */ unsigned nxt; /* right sibling, free list for root */ unsigned short ent; /* # entries */ unsigned short stk; /* offset of keyspace stack */ Dict dct[1]; } Block; #define DICTOFF ((char *)&((Block*)0)->dct - (char*)((Block*)0)) #define GETENT(b, i) (((unsigned char *)(b)) \ + ((b)->lev ? (b)->dct[i].pos : LEAFPOS((b)->dct+(i)))) /* LAYOUT: dct is an array of dictionary entries in increasing key order stk is growing downwards from block end towards dict position of entry i is b->stk <= LEAFPOS(b->dct+i) < dx->t/lsz decreasing on i (i.e. stack space is in reverse ordering) at LEAFPOS(b->dct+i), there are b->dct[i].kln bytes key for a leaf entry, the key is followed by LEAFVLN(b->dct+i) (between 0..2047 incl.) values of dx->vsz (between 8..23 incl) each for a fork entry, the key is followed by a single 4 bytes block number (vln = 0) or vln pairs of qualifying values (vsz each) and block numbers (without the key compression flag to be implemented, only vln=1 is supported) SIZES: block size >= 512 block base overhead DICTOFF is 16 bytes avail size >= 496 Dict entry is 4 bytes max. used stack space per entry is: - fork: kln+4+dx->vsz <= 255+4+255 = 514 - new leaf entry (one value): kln+dx->vsz <= 510 max total space for a new entry incl. 4 bytes dict is 518. For leaves of minimal block size 512, the sum of ksz+vsz must not exceed 492=512-16-4, so that each block can hold at least one entry. Forks are at least 4K and thus can hold min 7 entries. Even in pathological cases with max kln and vsz and all forks split to the worst, a level's fork factor does not drop below 4. */ enum { QMAXLEAF = 0x2000, /* max leaf size -- 8K fixed !!! */ QMAXFORK = 1< 8 Bytes) */ /* blocksize for files */ /* #define MERGEBLK 0x20000 = 128K buffer, 48-way merge #define MERGEBLK 0x3000 = 12K buffer, 512-way merge 12K blocks perform quite well and allow for 512-way merge, so we never have a second pass for 31-bit addressable files */ #define MERGEBLK 0x3000 #define MERGECHK (SORTMEM/MERGEBLK) /* initial chunk size */ /* By organizing files in MERGEBLKs, so that entries don't span blocks, we may be wasting up to one full entry (256 bytes) per block: if we have only exactly the space for one such longest entry, use one terminating 0 byte instead. */ #define SORTUSE (MERGECHK*(MERGEBLK-256)) typedef struct { unsigned char *keyb[SORTKEYS]; unsigned char memb[SORTMEM]; } SortBuf; typedef unsigned char MergeBlk[MERGEBLK]; /* n-way merge #define MERGEWAY (sizeof (SortBuf)/sizeof (MergeBlk)) */ #ifndef MERGEWAY /* define possible max */ # define MERGEWAY ((4*SORTKEYS+SORTMEM)/MERGEBLK) /* as pre-pro constant */ #endif #if 0x80000000UL > (MERGEWAY*SORTMEM) # define MERGEMULTIPASS /* else (512-way merge on 4MB) we can do 2GB in one pass */ #endif typedef struct QLoad { unsigned got; /* current for this key */ unsigned tot; /* total for this key */ unsigned let; /* stats */ unsigned keys; unsigned keyl; unsigned maxl; unsigned vals; unsigned maxv; unsigned span; unsigned free; /* load buffer */ Leaf cur; Key key; Val val[QENTMAX]; /* sort buffer */ file fil; unsigned blks; unsigned char **nkey; unsigned char *nmem; #if 0 /* hard to measure ... */ # define ALIGN4K __attribute__ ((aligned (0x1000))) #else # define ALIGN4K #endif union { SortBuf s; MergeBlk m[MERGEWAY]; /* mergebuf */ } buf ALIGN4K; MergeBlk mout; /* only used temporary, but don't want to kill stack */ } QLoad; typedef struct { /* structure filled by findkey */ Qdx *dx; /* the tree */ Key *key; /* the key searched */ unsigned n[QLEVELS]; /* block numbers of path */ unsigned short e[QLEVELS]; /* entries in path with match flag */ } FindKey; typedef struct { unsigned key; unsigned cmp; unsigned fget; unsigned lget; } Stat; /* buffer to print a value to */ typedef char PrVal[64]; /* ************************************************************ private data */ static Stat stat; /* ************************************************************ private functions */ /* bail out */ #define DONE( args ) do { ret = eRr args; goto done; } while (0) #ifdef CPU_NEED_ALIGN static unsigned GETINT ( void *m ) /* bus error safe version of *(int*) */ { unsigned l; memcpy( &l, m, 4 ); return l; } static void SETINT ( void *m, unsigned i ) { memcpy( m, &i, 4 ); } #else #define GETINT( m ) (*(unsigned*)(m)) #define SETINT( m, i ) (*(unsigned*)(m) = (i)) #endif void qMkVal ( Val *v, Ptr *p, unsigned char typ ) { unsigned char *c = v->byt; /* TODO: may macros to address the bytes in the ints would be faster */ if ( !(QDX_FULTXT & typ) ) /* leading tag */ switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: *c++ = (unsigned char) (p->tag >> 8); case QDX_TAG1: *c++ = (unsigned char) p->tag; } switch ( QDX_RIDMSK&typ ) { case QDX_RID6: *c++ = (unsigned char) (p->ext >> 8); case QDX_RID5: *c++ = (unsigned char) p->ext; case QDX_RID4: *c++ = (unsigned char) (p->rid >> 24); case QDX_RID3: *c++ = (unsigned char) (p->rid >> 16); *c++ = (unsigned char) (p->rid >> 8); *c++ = (unsigned char) p->rid; } if ( QDX_FULTXT & typ ) switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: *c++ = (unsigned char) (p->tag >> 8); case QDX_TAG1: *c++ = (unsigned char) p->tag; } switch ( QDX_POSMSK&typ ) { case 4: *c++ = (unsigned char) (p->pos >> 24); case 3: *c++ = (unsigned char) (p->pos >> 16); case 2: *c++ = (unsigned char) (p->pos >> 8); case 1: *c++ = (unsigned char) p->pos; } v->len = c - v->byt; } /* qMkVal */ int qRdKey ( Qdx *qdx, char *plain, int l, Key *key ) { if (!qdx->cdx) { if ( l > key->len ) l = key->len; memcpy(plain, key->byt, l); } else l = cDec(qdx->cdx, (unsigned char*)plain, l, key); return l; } /* qRdKey */ void qMkKey ( Qdx *qdx, Key *key, char *b, int l ) { if (!qdx->cdx) { if ( l > qdx->ksz ) l = qdx->ksz; memcpy(key->byt, b, key->len = l); } else { key->len = qdx->ksz; cEnc(qdx->cdx, key, (unsigned char*)b, l, 0); } } /* qMkKey */ int qMkKeyVal ( Qdx *qdx, Key *key, char *b, int l ) { Ptr p = {0,0,0,0}; char *t = memchr(b, '\t', l); if ( !t ) return -1; qMkKey(qdx, key, b, t - b); if ( 0 >= (l -= t-b+1) || !(t = memchr( b=t+1, '\t', l )) ) return -1; p.rid = (unsigned) a2i( b, t-b ); if ( 0 >= (l -= t-b+1) || !(t = memchr( b=t+1, '\t', l )) ) return -1; p.tag = (unsigned short) a2i( b, t-b ); if ( 0 >= (l -= t-b+1) || !(t = memchr( b=t+1, '\t', l )) ) return -1; p.pos = (unsigned) a2i( b, t-b ); if ( 0 < (l -= t-b+1) ) p.pos = p.pos << 16 | (unsigned short) a2i( t+1, l ); eRr(LOG_TRACE, "%'%b %d %d 0x%4x", key->len, key->byt, p.rid, p.tag, p.pos ); qMkVal( &key->val, &p, qdx->ptr ); return 0; } /* qMkKeyVal */ void qRdVal ( Ptr *p, const unsigned char *c, unsigned char typ ) { p->tag = p->ext = p->rid = p->pos = 0; if ( !(QDX_FULTXT & typ) ) /* leading tag */ switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: p->tag = (unsigned short)*c++ << 8; case QDX_TAG1: p->tag |= *c++; } switch ( QDX_RIDMSK&typ ) { case QDX_RID6: p->ext = (unsigned short)*c++ << 8; case QDX_RID5: p->ext |= *c++; case QDX_RID4: p->rid = (unsigned int)*c++ << 24; case QDX_RID3: p->rid |= (unsigned int)*c++ << 16; p->rid |= (unsigned int)*c++ << 8; p->rid |= *c++; } if ( QDX_FULTXT & typ ) switch ( QDX_TAGMSK&typ ) { case QDX_TAG2: p->tag = (unsigned short)*c++ << 8; case QDX_TAG1: p->tag |= *c++; } switch ( QDX_POSMSK&typ ) { case 4: p->pos = (unsigned int)*c++ << 24; case 3: p->pos |= (unsigned int)*c++ << 16; case 2: p->pos |= (unsigned short)*c++ << 8; case 1: p->pos |= *c++; } } /* qRdVal */ static char *prval ( PrVal buf, unsigned char *c, unsigned typ ) { char *d = buf; Ptr p; if ( ! typ ) { buf[0] = '#'; buf[1] = 0; return buf; } qRdVal( &p, c, typ ); if ( p.ext ) { d += u2a( d, p.ext ); *d++ = '.'; } d += u2a( d, p.rid ); *d++ = '.'; d += u2a( d, p.tag ); *d++ = '.'; d += u2a( d, p.pos ); *d++ = '.'; *d = 0; return buf; } /* prval */ #define PRVAL( buf, v ) prval( buf, v.byt, dx->ptr ) #define QLOCK(n,l) (void)FLOCK(dx->mqd, n, l) #define QLOCKSH(n) (void)FLOCKSH(dx->mqd, n) #define QLOCKEX(n) (void)FLOCKEX(dx->mqd, n) #define QLOCKUN(n) (void)FLOCKUN(dx->mqd, n) #ifdef BUILD_SHMODE enum { LOCK_READ = FIL_BLOCK, LOCK_WRITE = FIL_BLOCK|FIL_WR }; #endif /* ************************************************************ block I/O */ static int chkblk ( Block *b, Qdx *dx ) { int i; unsigned short eod = DICTOFF + b->ent*sizeof(Dict), siz, pos, vln, key; unsigned char *base = (unsigned char*)b; unsigned len = b->lev ? dx->fln : dx->lln; unsigned bsz = b->lev ? env.psz : dx->lsz; unsigned end = bsz, tot = 0; if ( b->num > len || (b->num && b->nxt > len) ) return eRr(ERR_TRASH, "bad num %d / nxt %d on lev %d", b->num, b->nxt, b->lev ); if ( b->typ != (b->lev ? dx->ftp : dx->typ) || b->ksz != dx->ksz || b->ptr != (b->lev?dx->vsz:dx->ptr) ) return eRr(ERR_TRASH, "bad sig 0x%02x,%d,%d in block %d want 0x%02x,%d,%d", b->typ, b->ksz, b->ptr, b->num, b->lev ? dx->ftp : dx->typ, dx->ksz, (b->lev?dx->vsz:dx->ptr) ); if ( b->ent > bsz / QENTSIZE || b->stk > bsz || b->stk < eod ) return eRr(ERR_TRASH, "bad ent/stk %d,%d in block %d eod %d sz %d", b->ent, b->stk, b->num, eod, bsz ); for ( i=0; ient; i++ ) { Dict *d = &b->dct[i]; if ( b->lev ) { pos = d->pos; vln = d->vln; siz = vln ? vln*(dx->vsz+4) : 4; } else { pos = LEAFPOS(d); vln = LEAFVLN(d); siz = vln*dx->vsz; } siz += key = d->kln; if ( pos < b->stk || end != (unsigned)pos + siz ) return eRr(ERR_TRASH, "bad pos/vln/key %d/%d/%d in entry %d of block %d" " stack %d sz %d end %d expected %d", pos, vln, key, i, b->num, b->stk, siz, pos+siz, end ); if ( b->lev ) { unsigned lnk = GETINT( base + pos + key + vln*dx->vsz ); if ( lnk >= (1==b->lev ? dx->lln : dx->fln) ) return eRr(ERR_TRASH, "bad link %u in entry %d of block %d", lnk, i, b->num ); } #ifndef NDEBUG if ( i ) { /* not on first */ unsigned short op = b->lev ? d[-1].pos : LEAFPOS(d-1), okl = d[-1].kln; int mc = memcmp( base+pos, base+op, keylev && vln ) { /* compare values */ unsigned ovln = d[-1].vln; if ( ! ovln || 0 < (j = memcmp( base+pos+key, base+op+okl, dx->vsz )) ) goto ok; } return eRr(ERR_TRASH, "bad key order entry %d of block %d lev %d" " mc %d j %d key %d okl %d vln %d", i, b->num, b->lev, mc, j, key, okl, vln ); } ok: #endif tot += siz; end = pos; } /* for i (entries) */ if ( tot > bsz - b->stk ) return eRr(ERR_TRASH, "bad total entry size %d in block %d stack %d", tot, b->num, b->stk ); #ifndef NDEBUG if ( LOG_TRACE >= env.log ) { eRr(LOG_TRACE, "block %d lev %d sig 0x%02x,%d ent %d eod %d stk %d", b->num, b->lev, b->typ, b->ksz, b->ent, eod, b->stk ); for ( i=0; i<2; i++ ) { int j = i ? b->ent-1 : 0; Dict *d = &b->dct[j]; if ( i >= b->ent ) /* if b->ent is 0 or 1, log no or one key */ break; if ( b->lev ) { pos = d->pos; vln = d->vln; } else { pos = LEAFPOS(d); vln = LEAFVLN(d); } key = d->kln; /* start of value */ if ( b->lev ) eRr(LOG_TRACE, "key %d '%b'/%d -> %d", j, key, base+pos, vln, GETINT(base+pos+key+(vln?4:0)) ); else { Ptr p; qRdVal( &p, base+pos+key, dx->ptr ); eRr(LOG_TRACE, "key %d '%b'[%d] = %d[%d,%d]", j, key, base+pos, vln, p.rid, p.tag, p.pos ); } } } #endif return 0; } /* chkblk */ /* init a new leaf in the local buffer b (usually a stack based Leaf). increment dx's lln */ static void newleaf ( Block *b, Qdx *dx ) { memset(b, 0, dx->lsz); b->num = dx->lln++; b->typ = dx->typ; b->ksz = dx->ksz; b->ptr = dx->ptr; b->lev = 0; b->stk = dx->lsz; } /* newleaf */ /* init a new fork in the local buffer b (usually a stack based Fork). increment dx's fln TODO: for qSet, use variant checking for free list at root->nxt. */ static Block *newfork ( Block *b, Qdx *dx, unsigned char lev ) { memset(b, 0, env.psz); b->num = dx->fln++; b->typ = dx->ftp; b->ksz = dx->ksz; b->ptr = dx->vsz; b->lev = lev; b->stk = env.psz; #if 16 == CPU_PAGE_SHIFT /* possible short overflow on ia64 */ if ( !b->stk ) b->stk--; #endif return b; } /* newfork */ #ifdef CPU_BIG_ENDIAN static unsigned rvi ( unsigned i ) { unsigned r; ((char*)&r)[0] = ((char*)&i)[3]; ((char*)&r)[1] = ((char*)&i)[2]; ((char*)&r)[2] = ((char*)&i)[1]; ((char*)&r)[3] = ((char*)&i)[0]; return r; } static unsigned short rvs ( unsigned short i ) { unsigned short r; ((char*)&r)[0] = ((char*)&i)[1]; ((char*)&r)[1] = ((char*)&i)[0]; return r; } #define SWI( i ) b->i = rvi( b->i ) #define SWS( s ) b->s = rvs( b->s ) static void swapleaf ( Block *b ) /* leaves are little endian */ { SWI(num); SWI(nxt); SWS(ent); SWS(stk); } #else # define swapleaf(b) #endif /* writing a leaf will render the leaf scrambled on big endians */ static int wrtleaf ( Block *b, Qdx *dx ) { unsigned num = b->num; int got; if ( chkblk( b, dx ) ) /* don't write no shize */ return eRr(ERR_TRASH, "\tbefore writing dx block %d", b->num ); swapleaf(b); /* a leaf written is a leaf lost */ got = fPwrite(&dx->mqd, b, dx->lsz, num*dx->lsz ); eRr(LOG_TRACE, "writing leaf %u got %u", num, got ); if ( dx->lsz != (unsigned)got ) return eRr(LOG_SYSERR, "\twhen writing dx block %d", b->num ); return 0; } /* wrtleaf */ /* writing a fork may invalidate all mapped forks due to remap */ static int wrtfork ( Block *b, Qdx *dx ) { int got; if ( chkblk( b, dx ) ) /* don't write no shize */ return eRr(ERR_TRASH, "\tbefore writing dx block %d", b->num ); LOG_DBG( LOG_DEBUG, "writing fork %d/%d from %8x/%8x", b->num, dx->mqx.npg, b, dx->mqx.map); if ( dx->mqx.npg ) { /* have mmap -- not during qload */ if ( b->num < dx->mqx.npg ) { /* should be in map */ if ( (char*)b != dx->mqx.map + (b->num<num); if ( ENV_MSYNC & env.flg ) fMSync(&dx->mqx, b->num); return 0; } if ( b->num < dx->mqx.lim /* try to extend mapping */ && 0 < (got = fMap(&dx->mqx, b->num+1)) && b->num < (unsigned)got ) { eRr(LOG_VERBOSE, "remapped to %d at %8x", dx->mqx.npg, dx->mqx.map); memcpy(dx->mqx.map + (b->num<mqx, b->num); return 0; } eRr(LOG_WARN, "remap to %d failed", b->num+1); /* else write to file */ } got = fPwrite(&dx->mqx.fil, b, env.psz, b->num*env.psz ); eRr(LOG_TRACE, "writing fork %u got %u", b->num, got ); if ( env.psz != (unsigned)got ) return eRr(LOG_SYSERR, "\twhen writing dx block %d", b->num ); return 0; } /* wrtfork */ static int chkread ( Block *b, Qdx *dx, int got, unsigned bsz, unsigned num, unsigned char lev ) { if ( 0 > got || bsz != (unsigned)got ) return eRr(LOG_SYSERR, "\twhen reading dx block %d", num ); if ( b->num != num ) return eRr(ERR_TRASH, "bad num %d in block %d", b->num, num ); if ( lev && b->lev != lev ) return eRr(ERR_TRASH, "bad lev %d in block %d want %d", b->lev, b->num, lev ); return chkblk( b, dx ); } /* chkread */ /* read a leaf into the provided local buffer */ static Block *getleaf ( Block *b, Qdx *dx, unsigned num #ifdef BUILD_SHMODE , int lck # define GETLEAF(leaf, dx, num, lck) getleaf(leaf, dx, num, lck) #else # define GETLEAF(leaf, dx, num, lck) getleaf(leaf, dx, num) #endif ) { int got; #ifdef BUILD_SHMODE if ( lck ) QLOCK(num<<1, lck); #endif got = fPread(&dx->mqd, b, dx->lsz, num*dx->lsz); stat.lget++; #ifdef CPU_BIG_ENDIAN if ( 0 < got && dx->lsz == (unsigned)got ) swapleaf(b); #endif if ( chkread(b, dx, got, dx->lsz, num, 0) ) b = 0; #ifdef BUILD_SHMODE if ( LOCK_READ == lck ) QLOCKUN(num<<1); #endif return b; } /* getleaf */ /* pick a fork from cache or read into the provided local buffer */ static Block *getfork ( Block *b, Qdx *dx, unsigned num, unsigned char lev ) { int got = env.psz; stat.fget++; if ( num < dx->mqx.npg ) b = (Block*)(dx->mqx.map + num*env.psz); else got = fPread(&dx->mqx.fil, b, env.psz, num*env.psz); return chkread(b, dx, got, env.psz, num, lev) ? 0 : b; } /* getfork */ /** binsearch position for key in a leaf. position is either the index of the matching entry (0..b->ent-1) or the index (0..b->ent) where key should be inserted. @return position for key (0..b->ent), |yfied with QMATCH, if found */ static unsigned short keyleaf ( Block *b, Key *key ) { unsigned char *base = (unsigned char*)b; unsigned char *kbt = key->byt; unsigned char kln = key->len; unsigned short l = 0, r = b->ent; Dict *d = b->dct; /* binsearch invariant: key is < all entries i >= r (we do know relation at r) key is > all entries i < l (don't know relation at l) */ while ( l < r ) { /* invariant: l <= keypos < r */ unsigned short i = (l + r) >> 1; /* l <= i < r */ unsigned short pos = LEAFPOS(&d[i]); unsigned char cln = d[i].kln; int cmp = memcmp( kbt, base+pos, kln 0 ) l = i+1; else r = i; } return l; /* == r */ } /* keyleaf */ /* same for forks. somewhat more complex since we might have duplicate keys with different values. Must not use the _POS/_VLN macros. */ static unsigned short keyfork ( Block *b, Key *key, unsigned char vsz ) { unsigned char *base = (unsigned char*)b; unsigned char *kbt = key->byt; unsigned char kln = key->len; unsigned short l = 0, r = b->ent; Dict *d = b->dct; /* binsearch invariant: key is < all entries i >= r (we do know relation at r) key is > all entries i < l (don't know relation at l) */ while ( l < r ) { /* invariant: l <= keypos < r */ unsigned short i = (l + r) >> 1; /* l <= i < r */ unsigned short pos = d[i].pos; unsigned char cln = d[i].kln; int cmp = memcmp( kbt, base+pos, klnval.len ) /* no value */ return QMATCH | i; cmp = 1; } else if ( ! key->val.len ) cmp = -1; else if ( !(cmp = memcmp( key->val.byt, base+pos+cln, key->val.len < vsz ? key->val.len : vsz)) && !(cmp = (int) key->val.len - vsz) ) return QMATCH | i; } if ( cmp > 0 ) l = i+1; else r = i; } return l; /* == r */ } /* keyfork */ /** find position for (leaf) val. search v in array b of n values of size sz @return position for value (0..n), |yfied with QMATCH, if found */ static unsigned short valpos ( unsigned char *b, unsigned short n, unsigned char sz, unsigned char *v ) { unsigned short l = 0, r = n; /* binsearch invariant: v is < all values i >= r (we do know relation at r) v is > all values i < l (don't know relation at l) */ while ( l < r ) { /* invariant: l <= valpos < r */ unsigned short i = (l + r) >> 1; /* l <= i < r */ int cmp = memcmp( v, b+i*sz, sz ); if ( ! cmp ) return QMATCH | i; if ( cmp > 0 ) l = i+1; else r = i; } return l; /* == r */ } /* valpos */ #define TRACEKEY( l, f ) (void)(f) /* find and fetch key's leaf into the provided buffer. */ static int findkey ( Block *leaf, FindKey *f #ifdef BUILD_SHMODE , int lck # define FINDKEY(leaf, f, lck) findkey(leaf, f, lck) #else # define FINDKEY(leaf, f, lck) findkey(leaf, f) #endif ) { #ifdef BUILD_SHMODE Qdx *dx = f->dx; /* for the lock macros */ #endif Fork fork[2]; /* buffer */ Leaf lbuf; int i, ret=-1; unsigned char *pkey = 0, pkln = 0, pvln = 0; Block *b; QLOCKSH(1); b = getfork(&fork[0].b, f->dx, 0, 0); if ( !b ) DONE((ERR_TRASH, "no root")); if ( b->lev >= QLEVELS ) DONE((ERR_TRASH, "tree is too deep: %u", b->lev)); stat.key++; for ( i=QLEVELS; i--; ) /* paranoia */ f->n[i] = f->e[i] = 0; for ( i=b->lev;; ) { unsigned short e; unsigned n; Dict *d; int tog = 0; f->n[i] = b->num; e = i ? keyfork(b, f->key, f->dx->vsz) : keyleaf(b, f->key); LOG_DBG( LOG_DEBUG, "keypos %4x", e ); /* here goes the Lehmann/Yao race detection: If we ran on right boundary, follow right link */ while ( !(QMATCH & e) && b->ent == e && b->nxt && i < f->dx->dpt ) { Block *s; unsigned se; /* peek next block, toggling buffers */ if ( i ) { /* TODO: actually we should never see this in forks as long as we lock the complete fork file */ s = getfork(&fork[(++tog)&1].b, f->dx, b->nxt, i); se = keyfork(s, f->key, f->dx->vsz); } else { s = GETLEAF((++tog)&1?&lbuf.b:leaf, f->dx, b->nxt, lck); se = keyleaf(s, f->key); #ifdef BUILD_SHMODE if ( LOCK_WRITE == lck ) QLOCKUN(se ? b->num : s->num); #endif } if ( !se ) /* lower than first */ break; LOG_DBG( LOG_DEBUG, "LYao %d -> %d lev %d for '%b' got 0x%4x", b->num, s->num, i, f->key->len, f->key->byt, se ); /* trade b for s */ f->n[i] = (b = s)->num; e = se; } f->e[i] = e; if ( ! i ) { if ( b != leaf ) memcpy(leaf, b, f->dx->lsz); return 0; /* keep the lock */ } if ( !(QMATCH & e) ) { if ( e ) f->e[i] = --e; else { TRACEKEY( ERR_TRASH, f ); return eRr(ERR_IDIOT, "ran on left bound of block %u ent %u", b->num, b->ent ); } } d = b->dct + (QENTMASK & e); /* direct access ok in fork */ pkey = ((unsigned char*)b)+d->pos; pkln = d->kln; pvln = d->vln ? f->dx->vsz : 0; n = GETINT( pkey+pkln+pvln ); if ( --i ) b = getfork(&fork[0].b, f->dx, n, i); else { QLOCKUN(1); b = GETLEAF(leaf, f->dx, n, lck); /* read 1st leaf */ /* continue only to do the LY */ } } done: QLOCKUN(1); return ret; /* go away */ } /* findkey */ /* ************************************************************ public functions */ int qRemake ( Qdx *dx ) { Key last; Bloke child; Fork fork; unsigned char lev = 0; /* the current child level to be indexed */ unsigned left = 0; /* block on child level */ unsigned iblk = 0; unsigned ikey = 0; unsigned ikeyl = 0; unsigned ipfx = 0; unsigned ispn = 0; fTrunc(&dx->mqx.fil, 0); fMap(&dx->mqx, 0); dx->fln = 1; /* pretend we had a root */ /* build tree */ for (dx->fln = 1;;lev++) { /* levels */ unsigned len = 0; /* current is n'th block on level */ Block *b = 0; unsigned char *base = 0; /* == b */ unsigned have = 0; /* room in current block b */ Dict *d = 0; /* of current block b */ unsigned next = left; /* child */ left = dx->fln; /* get child c and create child pointer in block b */ do { unsigned char vln = 0; /* actual byte length: dx->vsz or 0 */ unsigned char *key; unsigned char kln; unsigned need; Block *c = lev ? getfork(&child.b, dx, next, lev) : GETLEAF(&child.b, dx, next, 0);/* must have full ex lock */ if ( ! c ) return eRr(ERR_NOMEM, "no child block" ); next = c->nxt; if ( ! c->ent ) /* OOPS empty node */ continue; key = GETENT(c, 0); kln = c->dct[0].kln; if ( ! b ) /* the leftmost child on this level */ kln = 0; /* always 0 key on left */ else if ( lev ) /* the child is a fork -- have to carry vals ... */ vln = c->dct[0].vln ? dx->vsz : 0; /* ... if the child starts with */ else if ( last.len == kln /* compare key to last of prev. ... */ && !memcmp( last.byt, key, kln ) /* ... which was saved in last */ ) { /* key spans blocks */ LOG_DBG( LOG_DEBUG, "found span key '%b' child %u", kln, key, c->num ); vln = dx->vsz; /* qualify with value */ } else { unsigned char x = 0; /* find prefix key */ if ( kln > last.len ) kln = last.len; while ( x < kln && key[x] == last.byt[x] ) x++; if ( x < kln ) /* they differ at [x] */ kln = x+1; else /* equal on the shorter length */ kln++; if ( kln > c->dct[0].kln ) return eRr(ERR_TRASH, "bad key '%b' on level %u matching previous of length %u", c->dct[0].kln, key, lev, last.len ); LOG_DBG( LOG_DEBUG, "prefix on key '%b' child %u save %u", kln, key, c->num, c->dct[0].kln - kln ); ipfx += c->dct[0].kln - kln; } if ( vln ) ispn++; ikeyl += kln; need = sizeof(Dict) + kln + vln + 4; if ( need > have ) { if ( b ) { eRr(LOG_TRACE, "level %u block %u is full %u ent stack %u", lev, b->num, b->ent, b->stk ); b->nxt = dx->fln; /* link */ if ( wrtfork(b, dx) ) goto nowrite; len++; } b = newfork(&fork.b, dx, lev+1); base = (unsigned char*)b; have = env.psz - DICTOFF; /* - ql->let; TODO: should we ??? */ d = b->dct; len++; iblk++; } SETINT( base+(b->stk -= 4), c->num ); d->kln = kln; d->vln = vln ? 1 : 0; if ( kln += vln ) /* key + consecutive qualifying value */ memcpy( base+(b->stk -= kln), key, kln ); d->pos = b->stk; d++; b->ent++; ikey++; have -= need; /* save the last dance for me */ memcpy(last.byt, GETENT(c, c->ent-1), last.len = c->dct[c->ent-1].kln); } while (next); if ( !len ) return eRr(ERR_TRASH, "level %u was empty", lev ); /* close level */ eRr(LOG_INFO, "closing level %u with %u blocks %u..%u", lev, len, left, b->num ); if ( 1 == len ) b->num = 0; /* make it the root */ if ( wrtfork(b, dx) ) goto nowrite; if ( 1 == len ) { dx->dpt = b->lev; break; } } #define PM(a,b) (unsigned)(a*LULU(1000)/b) /* per mille */ if (ikey && ikeyl) /* divisors */ eRr(LOG_INFO, "\tused %u fork blocks %u keys %u bytes avg %d\n" "\t%u #spans (%d/oo) %u pfx (- %d/oo)", iblk, ikey, ikeyl, ikeyl/ikey, ispn, PM(ispn*dx->vsz,ikeyl), ipfx, PM(ipfx,(ipfx+ikeyl)) ); dx->fln--; /* counted one over */ fMap(&dx->mqx, dx->fln); return 0; nowrite: dx->fln = 0; return eRr(LOG_SYSERR, "\twhen writing tree block" ); } /* qRemake */ static int load ( Qdx *dx, unsigned char *key ) { int cmp = 0, keyl = 0, l; QLoad *ql = dx->qld; if (key) { keyl = *key - dx->vsz; l = keyl < ql->key.len ? keyl : ql->key.len; /* min */ if ( !(cmp = memcmp(key+1, ql->key.byt, l)) ) cmp = keyl - ql->key.len; if (0 > cmp) return eRr(LOG_ERROR, "qload key '%b' < last '%b'", keyl, key+1, ql->key.len, ql->key.byt ); } /* flush ? */ while ( !key /* end */ || QENTMAX == ql->got /* full */ || cmp /* next key */ ) { Block *b = &ql->cur.b; unsigned have = b->stk - DICTOFF - b->ent*sizeof(Dict) - ql->let; unsigned need = sizeof(Dict) + ql->key.len; unsigned want = need + ql->got * dx->vsz; unsigned take = ql->got; if ( have < want ) { /* you can't always get ... */ if ( have <= need ) take = 0; else { take = (have - need) / dx->vsz; if ( 256 > take * dx->vsz ) take = 0; } } LOG_DBG( LOG_DEBUG, "qload '%b' got %u need %u want %u have %u take %u", ql->key.len, ql->key.byt, ql->got, need, want, have, take ); /* we have enough room in b for take entries */ if ( take ) { unsigned char *base = (unsigned char*)b; unsigned char *s = base + b->stk; int i = take; Dict *d = &b->dct[ b->ent++ ]; while ( i-- ) memcpy( s -= dx->vsz, ql->val[i].byt, dx->vsz ); memcpy( s -= ql->key.len, ql->key.byt, ql->key.len ); b->stk = (unsigned short) (s - base); LEAFSETPV( d, b->stk, take ); d->kln = ql->key.len; if ( ql->got > take ) memcpy( ql->val, ql->val+take, sizeof(ql->val[0])*(ql->got - take) ); ql->got -= take; } if ( ! ql->got ) break; if ( take ) /* key spans blocks */ ql->span++; ql->free += b->stk - DICTOFF - b->ent*sizeof(Dict); /* wasted */ b->nxt = dx->lln; /* link */ LOG_DBG( LOG_DEBUG, "qload: %u entries left, getting block %u", ql->got, dx->lln ); if ( wrtleaf(b, dx) ) return eRr(LOG_SYSERR, "\twhen writing qload block" ); /* so couldn't flush all, thus we need another block */ eRr(LOG_TRACE, "new qload leaf %d", dx->lln); newleaf(b = &ql->cur.b, dx); /* will continue on end or keychange */ LOG_DBG( LOG_TRACE, "new block %u has %u ent %u stk, will keep %u", b->num, b->ent, b->stk, ql->let ); } if (cmp || !key) /* done with key: stats */ if (ql->maxv < ql->tot) ql->maxv = ql->tot; if (key) { if (cmp) { /* set new key */ if (ql->got) return eRr(ERR_IDIOT, "qload not empty on new key"); ql->tot = 0; memcpy(ql->key.byt, key+1, ql->key.len = keyl); ql->keys++; ql->keyl += keyl; if (ql->maxl < (unsigned)keyl) ql->maxl = keyl; } /* add new val */ /* LOG_HEX( key->val.byt, dx->vsz ); */ memcpy(ql->val[ ql->got++ ].byt, key+1+keyl, dx->vsz); ql->vals++; ql->tot++; } else { /* stats */ unsigned size = dx->lln * dx->lsz; unsigned sval = ql->vals * dx->vsz; unsigned sove = dx->lln * DICTOFF + sizeof(Dict)*(ql->keys+ql->span); if (size && ql->keys) /* divisors */ eRr(LOG_INFO, "end qload:\n" "\t#blks %u total len %u free %u (%d/oo) overhead %u (%d/oo)\n" "\t#keys %u total len %u (%d/oo) avg %d max %u\n" "\t#vals %u total len %u (%d/oo) avg %d max %u spans %u", dx->lln, size, ql->free, PM(ql->free,size), sove, PM(sove,size), ql->keys, ql->keyl, PM(ql->keyl,size), ql->keyl/ql->keys, ql->maxl, ql->vals, sval, PM(sval,size), ql->vals/ql->keys, ql->maxv, ql->span ); wrtleaf(&ql->cur.b, dx); } return 0; } /* load */ static int ncomp; #ifdef SORT_Q static int compar (const void *pa, const void *pb) { unsigned char *a = *(unsigned char **)pa; unsigned char *b = *(unsigned char **)pb; int l = *a, cmp; if (l > *b) l = *b; ncomp++; return (cmp = memcmp(a+1, b+1, l)) ? cmp : (int)*a - (int)*b; } #endif static OPT_INLINE int keyGt (unsigned char *a, unsigned char *b) { int l = *a, cmp; if (l > *b) l = *b; ncomp++; return 0 < (cmp = memcmp(a+1, b+1, l)) || (!cmp && *a > *b); } static int sort (Qdx *dx, unsigned char **key, int len) { unsigned char rev[SORTKEYS]; unsigned char *tmp; int i, ret = 0; lolo tm; tUpd(&tm); if (2 > len) goto cp; #ifdef SORT_Q qsort(key, len, sizeof key[0], compar); #else # define SORT_WH /* basic weak-heap-sort (Dutton 93) just like rSort; c.f. uti.c ``On the Performance of WEAK-HEAPSORT http://www.informatik.uni-freiburg.de/~edelkamp/publications/Weakheap.pdf for tuned versions see http://www.jea.acm.org/2002/EdelkampHeapsort/ and ``Pushing the Limits in Sequential Sorting http://www.informatik.uni-freiburg.de/~edelkamp/publications/Improvements.pdf */ memset(rev, 0, len); /* weak-heap-forestify every node roots a weak heap AND is <= root (especially the leftist independent roots are <= root) */ for ( i=len; --i; ) { int g = i; /* granma is 1st left i or root, if i is totally leftist ;) */ while (!(1&g)) /* we're left child */ g >>= 1; /* shift granma right :( */ g >>= 1; if (!keyGt(key[i], key[g])) /* merge: swap if g < i */ continue; tmp = key[i]; key[i] = key[g]; key[g] = tmp; /* swap */ rev[i] = 1; } /* re-weak-heap-forestify */ for ( i=len; --i>1; ) { int j=1, left; /* rev[i] = 0; i's REVBIT no longer needed */ /* follow path of left childs (independent roots) */ while (i > (left = (j<<1) + rev[j])) j = left; /* put max of them and i (new root) at i */ do { if (!keyGt(key[j], key[i])) /* merge: swap if i < j */ continue; tmp = key[i]; key[i] = key[j]; key[j] = tmp; /* swap */ rev[j] ^= 1; /* toggle revbit */ } while (j >>= 1); } /* rev[1] = 0; 2... have been cleared */ key[len] = key[0]; /* initial root still untouched biggest */ #endif #ifndef SORT_WH key--; #endif eRr(LOG_VERBOSE, "%d sorted %d keys with %d comparisions -- loading ...", tUpd(&tm), len, ncomp); cp: /* finally cp to dest */ if (FIL_NONE == dx->qld->fil) { /* internal -- load */ unsigned char **e = key+len+1; while (++key != e) if ((ret = load(dx, *key))) goto err; eRr(LOG_VERBOSE, "%d loaded", tUpd(&tm)); } else { /* external -- write MERGECHK blks */ unsigned char **e = key+len+1; unsigned char *out = dx->qld->mout, *end = out+MERGEBLK, *dst = out; unsigned blks = 0; while (++key != e) { unsigned l = 1+**key; if (end <= dst+l) { *dst = 0; if (0 > (ret = fWrite(&dx->qld->fil, out, MERGEBLK))) goto err; dst = out; if (MERGECHK == ++blks) return eRr(ERR_IDIOT, "CHUNK!"); } memcpy(dst, *key, l); dst += l; } do { *dst = 0; fWrite(&dx->qld->fil, out, MERGEBLK); dst = out; } while (MERGECHK != ++blks); eRr(LOG_VERBOSE, "%d wrote %d blocks ...", tUpd(&tm), blks); dx->qld->blks += MERGECHK; ret = 0; } err: return ret; } /* sort */ static int merge (Qdx *dx, QLoad *ql) { #ifndef MERGEMULTIPASS # define chunk MERGECHK #else unsigned char *const out = ql->mout, *const end = out+MERGEBLK; unsigned chunk = MERGECHK, pass = 1; file fil2 = FIL_NONE; #endif lolo tm; int ret = 0; eRr(LOG_VERBOSE, "%d-way merging %d blocks in %d chunks of %d ...", MERGEWAY, ql->blks, (ql->blks+MERGECHK-1)/MERGECHK, MERGECHK); tUpd(&tm); ncomp = 0; #ifndef MERGEMULTIPASS { /* single pass */ #else if (ql->blks > MERGECHK*MERGEWAY) /* multipass */ fOpen(&fil2, 0, 0); for (;; pass++) { #endif unsigned blkno = 0; #ifndef MERGEMULTIPASS # define inf &ql->fil { /* single run in single pass */ #else unsigned nchunk = chunk*MERGEWAY; /* lastrun, if >= ql->blks */ int lastpass = nchunk >= ql->blks, run = 1; file *inf, *outf; if (1 & pass) { inf = &ql->fil; outf = &fil2; } else { inf = &fil2; outf = &ql->fil; } if (!lastpass) fSeek(outf, 0); while (blkno < ql->blks) { /* merge MERGEWAY chunks of length chunk to n chunks of size nchunk */ /* only one run in last pass */ unsigned char *dst = out; unsigned blks = 0; #endif #ifdef MERGEDEBUG unsigned char lastkey[256]; #endif struct { unsigned char *blk; /* merge blk */ unsigned char *key; /* current key */ unsigned nxt; /* next block no */ unsigned end; /* end */ } in[MERGEWAY], *nxt[MERGEWAY], *tmp; unsigned ways = 0, i; for (ways=0; ways= ql->blks) break; in[ways].key = in[ways].blk = ql->buf.m[ways]; if (MERGEBLK != (ret = fPread(inf, in[ways].blk, MERGEBLK, blkno*MERGEBLK))) goto done; in[ways].nxt = blkno+1; if (ql->blks < (blkno += chunk)) blkno = ql->blks; in[ways].end = blkno; /* insert */ for (i=0; ikey, in[ways].key); ) i++; if (i < ways) memmove(nxt+i+1, nxt+i, (ways-i)*sizeof nxt[0]); nxt[i] = in+ways; } #ifndef MERGEMULTIPASS eRr(LOG_VERBOSE, "%d prepared %d blocks %d ways", tUpd(&tm), blkno, ways); #else eRr(LOG_VERBOSE, "%d prepared %d/%d blocks %d ways run %d of pass %d%s", tUpd(&tm), blkno, ql->blks, ways, run, pass, lastpass?" (last)":""); #endif #ifdef MERGEDEBUG lastkey[0] = 0; for (i=0; ikey, nxt[i]->key+1); #endif for (;;) { /* merge */ int len = *nxt[0]->key + 1; /* stuff away 1st key */ #ifdef MERGEDEBUG if (keyGt(lastkey, nxt[0]->key)) { ret = eRr(ERR_IDIOT, "'%b' > '%b' way %d", *lastkey, lastkey+1, *nxt[0]->key, nxt[0]->key+1, nxt[0]-in); goto done; } memcpy(lastkey, nxt[0]->key, len); #endif #ifdef MERGEMULTIPASS if (lastpass) { #endif if ((ret = load(dx, nxt[0]->key))) goto done; #ifdef MERGEMULTIPASS } else { if (len >= end - dst) { /* flush out block */ # ifdef MERGEDEBUG eRr(LOG_VERBOSE, "out blk %d len %d %b", blks, dst-out, *out, out+1); # endif *dst = 0; /* terminate */ if (MERGEBLK != (ret = fWrite(outf, dst = out, MERGEBLK))) goto done; blks++; } memcpy(dst, nxt[0]->key, len); dst += len; } #endif /* reload nxt[0] */ nxt[0]->key += len; if (!*nxt[0]->key) { /* eob */ if (nxt[0]->nxt != nxt[0]->end) { if (MERGEBLK != (ret = fPread(inf, nxt[0]->key = nxt[0]->blk, MERGEBLK, nxt[0]->nxt++ * MERGEBLK))) goto done; } if (!*nxt[0]->key) { /* done with this stream */ if (!--ways) break; memmove(nxt, nxt+1, ways*sizeof nxt[0]); continue; } } #ifdef MERGEDEBUG if (keyGt(lastkey, nxt[0]->key)) { ret = eRr(ERR_IDIOT, "'%b' > '%b' in blk %d off %d way %d", *lastkey, lastkey+1, *nxt[0]->key, nxt[0]->key+1, nxt[0]->nxt-1, nxt[0]->key-nxt[0]->blk, nxt[0]-in); goto done; } #endif if (1 == ways) continue; /* this may look like pretty brute force. However, in practice we rarely see reinsertions at all and get by with about 1.1 comparisions per key. With sorted input we have even less than 1 per key, due to the continue (last exit oneway) above. */ for (i=1; ikey, nxt[i]->key); ) i++; /* nxt[0]->key is not gt nxt[i]->key: move to --i */ if (--i) { tmp = nxt[0]; memmove(nxt, nxt+1, i*sizeof nxt[0]); nxt[i] = tmp; } } /* merge */ ret = 0; #ifndef MERGEMULTIPASS eRr(LOG_VERBOSE, "%d merged %d comparisions", tUpd(&tm), ncomp); } /* single run */ #else eRr(LOG_VERBOSE, "%d merged %d comparisions %d/%d blocks pass %d", tUpd(&tm), ncomp, blks, nchunk, pass); if (lastpass) goto done; if (nchunk <= blks) eRr(ERR_IDIOT, "wrote %d >= %d blks pass %d", blks, nchunk, pass); do { *dst = 0; fWrite(outf, out, MERGEBLK); dst = out; } while (nchunk != ++blks); run++; } /* for new chunks */ chunk = nchunk; #endif } /* pass */ done: #ifdef MERGEMULTIPASS if (FIL_NONE != fil2) fClose(&fil2); #endif return ret; } /* merge */ int qLoad ( Qdx *dx, Key *key ) { QLoad *ql = dx->qld; int ret, l = key->len+dx->vsz; if ( !ql ) { if (! key->val.len) return 0; /* double close ? */ ncomp = 0; if (ENV_EXCL!=env.wri || !(DX_WRITE & dx->flg)) return eRr(ERR_TRASH, "need excl write access to load index"); ql = dx->qld = (QLoad*)mAlloc(sizeof (QLoad)); if ( dx->let > 50 ) dx->let = 50; ql->let = dx->let * dx->lsz / 100; fTrunc(&dx->mqd, 0); dx->lln = 0; newleaf(&ql->cur.b, dx); ql->fil = FIL_NONE; ql->nkey = ql->buf.s.keyb; ql->nmem = ql->buf.s.memb; eRr(LOG_INFO, "loading %d/%dK mem, %d keys, %dK blocks in %d-chunks, %d-way merge" #ifdef MERGEMULTIPASS " (multipass)" #endif , SORTUSE>>10, SORTMEM>>10, SORTKEYS, MERGEBLK>>10, MERGECHK, MERGEWAY ); } if ( !key->val.len /* done */ || SORTKEYS-2 <= ql->nkey-ql->buf.s.keyb /* reserve 1 for WHS */ || SORTUSE <= ql->nmem-ql->buf.s.memb ) { if (key->val.len && FIL_NONE == ql->fil) fOpen(&ql->fil, 0, 0); if ((ret = sort(dx, ql->buf.s.keyb, ql->nkey - ql->buf.s.keyb)) || !key->val.len ) goto done; ql->nkey = ql->buf.s.keyb; ql->nmem = ql->buf.s.memb; } /* add key to sort buffer */ if (key->val.len == dx->vsz && l < 256) { *ql->nkey++ = ql->nmem; *ql->nmem = (unsigned char)l; memcpy(++ ql->nmem, key->byt, key->len); memcpy(ql->nmem += key->len, key->val.byt, dx->vsz); ql->nmem += dx->vsz; return 0; } ret = eRr(ERR_INVAL, "bad klen %d vlen %d", key->len, key->val.len); done: if (FIL_NONE != ql->fil) { if (!ret) ret = merge(dx, ql); fClose(&ql->fil); } if (!ret) { load(dx, 0); /* close last leaf */ qRemake(dx); /* build tree */ } mFree(dx->qld); dx->qld = 0; return ret; } /* qLoad */ int qLoadf ( Qdx *dx, file *f ) { Key key; lolo tm; FIL_DEFBUF(*f); memset( &key, 0, sizeof(key) ); key.val.len = dx->vsz; tUpd(&tm); while ( fGets(&fb) && fb.l ) { if ( qMkKeyVal(dx, &key, fb.p, fb.l) ) { eRr(LOG_VERBOSE, "bad key at line %d offset %d", fb.n, fb.o); continue; } if ( qLoad( dx, &key ) ) break; if ( !(fb.n & 0x3ff) ) eRr(LOG_VERBOSE, "%dK lines", fb.n >> 10 ); } eRr(LOG_INFO, "%d did %d lines %d bytes (%s)", tUpd(&tm), fb.n, fb.o, FIL_NONE == fb.f ? "eof" : "blank" ); *f = fb.f; key.val.len = 0; /* close it */ return qLoad(dx, &key); } /* qLoadf */ /* TODO: this function is an awful mess and needs complete rewrite, KR */ int qSet ( Qdx *dx, Key *key ) { Bloke blk[2]; Key up; Block * b = &blk[0].b; /* mainblock */ unsigned char * base = blk[0].base; int ret = 0; unsigned short e, i = 0; Block *s = 0; /* sibling */ Dict *d; unsigned char *sbase, *src, *v, *stk; FindKey f; int hadkey, split = 0, insibling = 0; unsigned need; unsigned sibnum = 0; unsigned char lev = 0; PrVal pv; #ifdef BUILD_SHMODE /* locks we hold on block, sibling, tree */ int lb=-1, ls=-1, lt=0; #endif f.dx = dx; f.key = key; if ( dx->ksz && key->len > dx->ksz ) key->len = dx->ksz; if ( FINDKEY(b, &f, LOCK_WRITE) ) { b = 0; DONE((ERR_TRASH, "could not find key")); } #ifdef BUILD_SHMODE lb = b->num; #endif e = (QENTMASK & f.e[0]); d = b->dct + e; if ( ! (hadkey = QMATCH & f.e[0]) ) /* new key */ need = key->len + sizeof(Dict) + dx->vsz; else { unsigned vln = LEAFVLN(d); v = base+LEAFPOS(d)+d->kln; if ( 1 == vln ) { /* check stopword -- value entirely zeroed */ int l = dx->vsz; unsigned char *C = v; while ( l-- && !*C++ ) /* huh huh, he said "C++" */ ; if ( -1 == l ) { LOG_DBG(LOG_VERBOSE, "key '%b' is a stopword", key->len, key->byt); return QST_STPWRD; } /* TODO: check QST_UNIKEY */ } i = valpos( v, vln, dx->vsz, key->val.byt ); if ( QMATCH & i ) { LOG_DBG( LOG_DEBUG, "adding key '%b'/%s: already there pos %u", key->len, key->byt, PRVAL( pv, key->val ), i & ~QMATCH ); return QST_FOUND; /* value already there */ } need = dx->vsz; } LOG_DBG( LOG_DEBUG, "adding key '%b'/%s in block %d had %d ent %d stk %d need %d", key->len, key->byt, PRVAL( pv, key->val ), b->num, hadkey, b->ent, b->stk, need ); if ( need > b->stk - DICTOFF - b->ent*sizeof(Dict) ) { /* now that's bad: not enough space create new block s, move last entries there starting from the end, entries (including the new one) are collected, until their combined size reaches more than half a block. If the combined size then is more than 2/3 a block, and the last entry alone takes more than 1/3 a block, it is split. TODO: check existing sibling for free space */ unsigned half = (dx->lsz - DICTOFF) >> 1; unsigned tot = 0; unsigned mv = b->ent; LOG_DBG( LOG_DEBUG, "splitting block %d need %d have %d (stk %d ent %d)", b->num, need, b->stk - DICTOFF - b->ent*sizeof(Dict), b->stk, b->ent ); if ( need > (dx->lsz - DICTOFF) / 3 ) DONE((ERR_IDIOT, "OOPS! excessive need for %d bytes", need)); /* figure out split position */ if ( !hadkey && b->ent == e ) { tot = need; insibling = !0; } d = b->dct+mv; while ( d--, mv-- ) { unsigned sz = sizeof(Dict)+d->kln+LEAFVLN(d)*dx->vsz; if ( half > (tot += sz) ) { /* sz fits */ if ( e == mv && !hadkey ) { /* check size of new entry. In case of hadkey, we don't bother for the dx->vsz bytes. */ if ( half < (tot + need) ) { if ( (half - tot) > need/2 ) { /* more balanced if we move */ tot += need; insibling = !0; } break; } tot += need; insibling = !0; } if ( mv ) continue; /* else croak on ! mv below */ } /* exceeded half -- maybe we move entry mv anyway */ if ( tot > (dx->lsz - DICTOFF)*2/3 ) { if ( tot-sz < (dx->lsz - DICTOFF)/3 ) split = !0; else { /* undo -- don't move this entry */ tot -= sz; d++; mv++; } } else if ( ! mv ) DONE((ERR_IDIOT, "OOPS! didn't find end tot %d bytes", tot)); break; } /* while mv-- */ if ( mv < e || (mv == e && hadkey && !split) ) insibling = !0; /* now mv is the first entry to move */ /* create new leaf block dx->lln */ #ifdef BUILD_SHMODE for (;;) { int size; QLOCKEX(dx->lln<<1); size = fSize( dx->mqd ); if ( size == (int)(dx->lln * dx->lsz) ) { ls = dx->lln; break; } QLOCKUN(dx->lln<<1); if ( size % dx->lsz || size < (int)(dx->lln * dx->lsz) ) DONE((ERR_TRASH, "bad mqd size 0x%08x", size)); dx->lln = size / dx->lsz; } /* have lock on new leaf block dx->lln */ #endif newleaf(s = &blk[1].b, dx); sbase = (unsigned char*)s; /* == blk[1].base */ LOG_DBG( LOG_DEBUG, "split: move %d of %d to new blk %d e %d tot %d insibling %d split %d", mv, b->ent, s->num, e, tot, insibling, split ); s->nxt = b->nxt; b->nxt = s->num; if ( split ) { /* now total includes full size of entry mv. if we included a new entry, it might even exceed lsz. calculate sizes for both blocks w/o mv's values. */ unsigned esz = sizeof(Dict)+d->kln; /* entry's base cost */ unsigned vln = LEAFVLN(d); unsigned keep, diff, mvval, kpval, mvsiz, pos = LEAFPOS(d); /* from currently used space, substract tot (including all values and possibly new key "need"), but add entry base cost. that should be the size remaining if we where moving all of the values but keep an entry and key. maybe slightly biased if need is dx->vsz (the hadkey case). */ keep = dx->lsz - b->stk + b->ent*sizeof(Dict) + need - tot + esz; tot -= vln * dx->vsz; /* undo entries */ diff = (keep > tot ? keep - tot : tot - keep) / dx->vsz; if ( diff > vln ) DONE((ERR_IDIOT, "OOPS! got diff %d vln %d entry %d tot %d keep %d", diff, vln, mv, tot, keep)); mvval = (vln - diff)/2; if ( keep > tot ) mvval += diff; kpval = vln - mvval; if ( ! mvval || ! kpval ) { LOG_DBG( LOG_WARN, "bad mv/kpval %d/%d keep %d tot %d vln %d diff %d", mvval, kpval, keep, tot, vln, diff ); if ( ! mvval ) kpval = vln - (mvval = 1); else mvval = vln - (kpval = 1); } tot += mvsiz = mvval*dx->vsz; keep += kpval*dx->vsz; src = base+pos; LOG_DBG( LOG_DEBUG, "split: entry %d '%b' mv %d of %d values, i %d keep %d tot %d", mv, d->kln, src, mvval, vln, i, keep, tot ); if ( keep > (dx->lsz-DICTOFF) || tot > (dx->lsz-DICTOFF) ) DONE((ERR_IDIOT, "OOPS! got diff %d vln %d mvval %d entry %d tot %d keep %d", diff, vln, mvval, mv, tot, keep)); /* SIGH -- do it */ memcpy( sbase + (s->stk -= mvsiz), src+d->kln+kpval*dx->vsz, mvsiz ); if ( e == mv && hadkey && i > kpval ) { insibling = !0; s->stk -= dx->vsz; i -= kpval; memmove( sbase+s->stk, sbase+s->stk+dx->vsz, i*dx->vsz ); memcpy( sbase+s->stk+i*dx->vsz, key->val.byt, dx->vsz ); mvval++; } memcpy( sbase + (s->stk -= d->kln), src, d->kln ); LEAFSETPVK( s->dct, s->stk, mvval, d->kln ); s->ent = 1; memmove( src + mvsiz, src, d->kln+kpval*dx->vsz ); pos += mvsiz; LEAFSETPV( d, pos, kpval ); d++; mv++; } /* now entries mv..b->ent are moved, mv > 0 */ b->stk = LEAFPOS(b->dct+mv-1); /* pre-adjust stack */ if ( mv < e ) { /* move entries mv..e-1 */ /* used stack space is from pos of entry e-1 to end of entry mv (might differ from pos of mv-1 due to split) */ unsigned from = LEAFPOS(b->dct+e-1), nent = e-mv, sz = LEAFPOS(b->dct+mv)+b->dct[mv].kln +LEAFVLN(b->dct+mv)*dx->vsz - from; int adj = (s->stk -= sz) - from; /* entry position adjustment */ memcpy( sbase + s->stk, base+from, sz ); memcpy( s->dct+s->ent, b->dct+mv, nent*sizeof(Dict) ); d = s->dct + s->ent; s->ent += nent; for ( ; nent--; d++ ) LEAFPOSADD( d, adj ); } if ( insibling && mv <= e ) { /* now mv or insert e, unless done on split */ d = b->dct + e; if ( hadkey ) { /* mv existing entry e >= mv, add new val */ /* i is valpos */ unsigned vln = LEAFVLN(d); src = base+LEAFPOS(d)+d->kln; if ( i < vln ) { unsigned sz = (vln - i)*dx->vsz; memcpy( sbase + (s->stk -= sz), src + i*dx->vsz, sz ); } memcpy( sbase + (s->stk -= dx->vsz), key->val.byt, dx->vsz ); if ( i ) { unsigned sz = i*dx->vsz; memcpy( sbase + (s->stk -= sz), src, sz ); } memcpy( sbase + (s->stk -= d->kln), base+LEAFPOS(d), d->kln ); LEAFSETPVK( s->dct + s->ent, s->stk, vln+1, d->kln ); e++; /* e is moved */ } else { memcpy( sbase + (s->stk -= dx->vsz), key->val.byt, dx->vsz ); memcpy( sbase + (s->stk -= key->len), key->byt, key->len ); LEAFSETPVK( s->dct + s->ent, s->stk, 1, key->len ); } s->ent++; } /* mv or insert e */ if ( e < b->ent && mv < b->ent ) { /* move entries max(e,mv) to end */ unsigned fst = mv < e ? e : mv; /* used stack space is from pos of entry b->ent-1 (the original b->stk) to end of entry fst (might differ from pos of fst-1 due to split) */ unsigned from = LEAFPOS(b->dct+b->ent-1), nent = b->ent-fst, sz = LEAFPOS(b->dct+fst)+b->dct[fst].kln +LEAFVLN(b->dct+fst)*dx->vsz - from; int adj = (s->stk -= sz) - from; /* entry position adjustment */ LOG_DBG( LOG_DEBUG, "split: entries %d..%d sz %d from %d to %d", fst, b->ent-1, sz, from, s->stk ); memcpy( sbase + s->stk, base+from, sz ); memcpy( s->dct+s->ent, b->dct+fst, nent*sizeof(Dict) ); d = s->dct + s->ent; s->ent += nent; for ( ; nent--; d++ ) { #if !defined(NDEBUG) unsigned p = LEAFPOS(d); #endif LEAFPOSADD( d, adj ); #if !defined(NDEBUG) LOG_DBG( LOG_DEBUG, "new ent %d p/v/k %d/%d/%d o %d", d - s->dct, LEAFPOS(d), LEAFVLN(d), d->kln, p ); #endif } } b->ent = mv; /* kill entries */ LOG_DBG( LOG_DEBUG, "split: blks %d/%d with %d/%d entries, s starts '%b'", b->num, s->num, b->ent, s->ent, s->dct[0].kln, sbase+LEAFPOS(s->dct) ); } /* new sibling */ if ( ! insibling ) { /* now we do have enough space */ d = b->dct + e; stk = base + b->stk; /* bottom of data stack */ if ( hadkey ) { /* insert value at i */ unsigned char *val = base + LEAFPOS(d)+d->kln+i*dx->vsz; need = dx->vsz; #if 0 LOG_DBG( LOG_DEBUG, "in entry %d '%b' val %d: memmove( %p %p %d) val %p need %d", e, d->kln, base+LEAFPOS(d), i, stk-need, stk, (val-stk)+need, val, need ); #endif memmove( stk - need, stk, (val - stk)+need ); memcpy( val - need, key->val.byt, dx->vsz ); LEAFVLNADD( d, 1 ); } else { /* insert key, value and dict entry */ /* take dict and data space from entry e */ unsigned char *nu; need = key->len + dx->vsz; if ( e == b->ent ) nu = base + b->stk - need; else { /* at end of next entry, i.e. end of block or beg of previous */ nu = base + LEAFPOS(d) + d->kln + LEAFVLN(d)*dx->vsz; memmove( stk - need, stk, nu - stk ); nu -= need; memmove( d+1, d, (b->ent - e)*sizeof(Dict) ); } memcpy( nu, key->byt, key->len ); memcpy( nu + key->len, key->val.byt, dx->vsz ); LEAFSETPV( d, nu-base, 1 ); d->kln = key->len; d++; e++; b->ent++; } /* need is set to the needed (i.e. additionally used) data space, e is the first dict entry to redirect need bytes lower */ b->stk -= need; for ( ; e < b->ent; e++, d++ ) LEAFPOSADD( d, -need ); } /* ! insibling */ if ( s ) { /* save upkey before writing, which may kill s */ unsigned char *e0 = ((unsigned char*)s) + LEAFPOS(s->dct); memcpy( up.byt, e0, up.len = s->dct[0].kln ); sibnum = s->num; if ( split ) memcpy( up.val.byt, e0+up.len, up.val.len = dx->vsz ); else up.val.len = 0; LOG_DBG( LOG_DEBUG, "split: had %d/%d key '%b' vln %d %s", b->num, s->num, up.len, up.byt, up.val.len, PRVAL( pv, up.val ) ); wrtleaf(s, dx); } wrtleaf(b, dx); /* WHOA -- now all that was just to care for the leaf level! Now walk up inserting pointers in parent forks while we split. */ #ifdef BUILD_SHMODE /* lock tree, unlock leaves */ QLOCKUN(lb<<1); lb = -1; if ( sibnum ) { QLOCKEX(1); /* get tree lock */ lt = 1; QLOCKUN(ls<<1); ls = -1; } #endif while ( sibnum ) { /* created sibling */ unsigned char *e0; int tog = 0; Block *ins; /* get parent */ lev++; b = getfork(&blk[0].b, dx, f.n[lev], lev); LOG_DBG( LOG_DEBUG, "split: got parent %d lev %d", b->num, b->lev ); e = keyfork(b, &up, dx->vsz); /* again L/Y link hunt */ while ( !(QMATCH & e) && b->ent == e && b->nxt && b->num ) { unsigned se; /* peek next block, toggling buffers */ s = getfork(&blk[(++tog)&1].b, dx, b->nxt, lev); se = keyfork(s, &up, dx->vsz); if ( !se ) /* lower than first */ break; LOG_DBG( LOG_DEBUG, "LYao %d -> %d lev %d for '%b' got 0x%4x", b->num, s->num, i, up.len, up.byt, se ); /* trade b for s */ b = s; e = se; } base = (unsigned char*)b; ins = b; s = 0; LOG_DBG( LOG_DEBUG, "split: got parent pos %4x", e ); if ( QMATCH & e ) /* !??? */ DONE((ERR_TRASH, "OOPS! found key in parent")); if ( ! e ) { TRACEKEY( ERR_TRASH, &f ); DONE((ERR_TRASH, "OOPS! parent insert at 0 lev %d", lev)); } need = sizeof(Dict) + up.len + 4 + up.val.len; if ( need > b->stk - DICTOFF - b->ent*sizeof(Dict) ) { unsigned half = ((env.psz - b->stk)+b->ent*sizeof(Dict)+need) / 2; unsigned mv = b->ent, tot = 0, nent, sz; int adj; if ( e == b->ent ) { tot = need; ins = 0; } for ( d = b->dct+mv; d--, mv--; ) { sz = sizeof(Dict) + d->kln + 4 + (d->vln ? dx->vsz : 0); if ( half > (tot += sz) ) { if ( e == mv ) { if ( half > tot+need || tot+need-half < half-tot ) { tot += need; ins = 0; /* new link goes to sibling */ } else break; if ( half <= tot ) break; } if ( mv ) continue; } if ( tot - half > sz/2 ) { /* don't move this */ tot -= sz; d++; mv++; } else if ( ! mv ) /* !??? */ mv = 1; break; } /* split here a little bit simpler */ s = newfork( b==&blk[0].b ? &blk[1].b : &blk[0].b, dx, lev ); s->nxt = b->nxt; b->nxt = s->num; /* straight move entries mv..b->ent-1 to sibling */ sbase = (unsigned char*)s; d = b->dct + mv; sz = b->dct[mv-1].pos - b->stk; nent = b->ent - mv; LOG_DBG( LOG_DEBUG, "split: parent lev %d split %d to new %d mv %d" " '%b'/%s nent %d sz %d", lev, b->num, s->num, mv, d->kln, base+d->pos, prval( pv, base+d->pos+d->kln, d->vln?dx->ptr:0 ), nent, sz ); memcpy( sbase + (s->stk -= sz), base + b->stk, sz ); adj = s->stk - b->stk; b->stk += sz; memcpy( s->dct, d, nent*sizeof(Dict) ); b->ent = mv; s->ent = nent; for ( d = s->dct; nent--; d++ ) d->pos += adj; if ( ! ins ) { ins = s; e -= mv; } if ( (ret = chkblk( s, dx )) ) break; } /* now insert link in block ins at pos e */ { unsigned isz = up.len + 4 + up.val.len, nent = ins->ent - e; unsigned char *ibase = (unsigned char*)ins; unsigned end = e ? ins->dct[e-1].pos : env.psz; LOG_DBG( LOG_DEBUG, "split: ins parent %d lev %d pos %d key '%b' -> %d vln %d %s", ins->num, ins->lev, e, up.len, up.byt, sibnum, up.val.len, PRVAL( pv, up.val ) ); memmove( ibase + ins->stk - isz, ibase + ins->stk, end - ins->stk ); ins->stk -= isz; SETINT(ibase + (end -= 4), sibnum); if ( up.val.len ) memcpy( ibase + (end -= up.val.len), up.val.byt, up.val.len ); memcpy( ibase + (end -= up.len), up.byt, up.len ); for ( d = ins->dct+ins->ent; nent--; d-- ) { *d = d[-1]; d->pos -= isz; } d->pos = end; d->vln = up.val.len ? 1 : 0; d->kln = up.len; ins->ent++; if ( (ret = chkblk( ins, dx )) ) break; } wrtfork(b, dx); if ( ! s ) /* not split - we're done */ break; /* save sibling's key */ sibnum = s->num; e0 = ((unsigned char*)s) + s->dct[0].pos; memcpy( up.byt, e0, up.len = s->dct[0].kln ); if ( s->dct[0].vln ) memcpy( up.val.byt, e0+up.len, up.val.len = dx->vsz ); else up.val.len = 0; LOG_DBG( LOG_DEBUG, "split: had %d/%d on lev %d key '%b' vln %d %s", b->num, sibnum, s->lev, up.len, up.byt, up.val.len, PRVAL( pv, up.val ) ); wrtfork(s, dx); if ( lev < dx->dpt ) continue; LOG_DBG( LOG_DEBUG, "root split for dx %8x!", dx); if ( b != &blk[0].b && b != &blk[1].b ) /* was in map -- could be gone */ b = getfork(&blk[0].b, dx, 0, lev); /* refetch */ LOG_DBG( LOG_DEBUG, "refetched root %8x!", b); /* move old root to new block, use sibling buffer (if any) */ s = newfork(b==&blk[0].b ? &blk[1].b : &blk[0].b, dx, lev); /* cp all but the num */ memcpy( ((unsigned char*)s)+4, ((unsigned char*)b)+4, env.psz-4 ); s->nxt = sibnum; /* and link */ wrtfork(s, dx); b->stk = env.psz; /* empty the root */ /* add two childs linking to old root and it's sibling */ base = (unsigned char*)b; SETINT( base + (b->stk -= 4), s->num ); /* link the old root */ b->dct[0].pos = b->stk; b->dct[0].vln = 0; b->dct[0].kln = 0; SETINT( base + (b->stk -= 4), sibnum ); /* link the sibling */ if ( ! up.val.len ) b->dct[1].vln = 0; else { memcpy( base + (b->stk -= dx->vsz), up.val.byt, dx->vsz ); b->dct[1].vln = 1; } memcpy( base + (b->stk -= up.len), up.byt, b->dct[1].kln = up.len ); b->dct[1].pos = b->stk; b->ent = 2; b->lev++; /* increase tree depth */ dx->dpt = b->lev; ret = wrtfork(b, dx); break; } done: #ifdef BUILD_SHMODE if ( -1 != lb ) QLOCKUN(lb<<1); if ( -1 != ls ) QLOCKUN(ls<<1); if ( lt ) QLOCKUN(1); #endif if ( 0 <= ret && b->lev && FINDKEY(&blk[0].b, &f, LOCK_READ) ) ret = eRr(ERR_TRASH, "could not find key after add" ); return ret; } /* qSet */ int qDel ( Qdx *dx, Key *key ) { Leaf leaf; Block * const b = &leaf.b; unsigned char * const base = leaf.base; int ret = 0; unsigned short e, n, i, mv; Dict *d; unsigned char *v; FindKey f; PrVal pv; f.dx = dx; f.key = key; LOG_DBG( LOG_DEBUG, "deleting key '%b'/%s", key->len, key->byt, PRVAL( pv, key->val ) ); if ( FINDKEY(b, &f, LOCK_WRITE) ) return eRr(ERR_TRASH, "could not find key"); if ( !(QMATCH & f.e[0]) ) /* key not found - ok */ goto done; LOG_DBG( LOG_DEBUG, "key found" ); e = (QENTMASK & f.e[0]); d = b->dct + e; v = base+LEAFPOS(d)+d->kln; /* start of values */ n = LEAFVLN(d); if ( ! n ) goto done; i = valpos( v, n, dx->vsz, key->val.byt ); LOG_DBG( LOG_DEBUG, "val %sfound pos %u of %u", (QMATCH & i) ? "" : "not ", i & ~QMATCH, n ); if ( !(QMATCH & i) ) /* val not found - ok */ goto done; i &= ~QMATCH; LOG_DBG( LOG_DEBUG, "del blk %d e %d i/n %d/%d", b->num, e, i, n ); if ( 1 < n ) { v += i*dx->vsz; /* end of stack area to move: start of this value */ mv = dx->vsz; /* amount to move: one value size */ LEAFVLNADD( d, -1 ); } else { /* sole value -- rm whole entry */ v = base+LEAFPOS(d); /* start of key */ mv = d->kln+dx->vsz; for ( i=b->ent - e - 1; i--; d++ ) /* move the i entries after d down */ *d = d[1]; d = b->dct + e; /* reset */ b->ent--; } memmove( base+b->stk+mv, base+b->stk, v-(base+b->stk) ); b->stk += mv; for ( ; e++ < b->ent; d++ ) LEAFPOSADD( d, mv ); wrtleaf(b, dx); done: QLOCKUN(b->num<<1); return ret; } /* qDel */ static int setKeyVal (QSet *qst, char *val, int len, int split) { Qdx *qdx = qst->qdx; char *e = val+len; Key key; int nope, done = 0; if (split && qst->pfx.len) split = 1+qst->pfx.len; /* don't split on 1st pfx.len bytes */ do { char buf[512], *b; int l; if (!split || qdx->cdx) l = e-val; else { char *s; while (valpfx.len) { memcpy(b = buf, qst->pfx.byt, qst->pfx.len); if (l > (int)sizeof buf-qst->pfx.len) l = sizeof buf-qst->pfx.len; memcpy(buf+qst->pfx.len, val, l); l += qst->pfx.len; } /* qMkKey(qst->qdx, &key, val, len): */ if (!qdx->cdx) memcpy(key.byt, b, key.len = lksz ? l : qdx->ksz); else { key.len = qdx->ksz; l = cEnc(qdx->cdx, &key, (unsigned char*)b, l, split); if ( split && (!key.len /* wordless */ || l <= qst->pfx.len) /* houston, we have a problem */ ) break; } qMkVal(&key.val, &qst->ptr, qdx->ptr); qst->ptr.pos++; eRr(LOG_VERBOSE, "%s '%.*s'->'%b' %u %u %u %u", qst->del ? "del" : "set", l, b, key.len, key.byt, qst->ptr.rid, qst->ptr.tag, qst->ptr.pos>>16, qst->ptr.pos&0xffff); nope = qst->del ? qDel(qdx, &key) : qSet(qdx, &key); if (!split) return nope; val += l - qst->pfx.len; if (!nope) done++; } while (val < e); return done; } /* qSetKeyVal */ int qSetKeyVal (QSet *qst, char *val, int len) { return setKeyVal(qst, val, len, 0); } int qSetKeyVals (QSet *qst, char *val, int len) { return setKeyVal(qst, val, len, 1); } /* qSetKeyVals */ int qLoop ( QLoop *self ) { Leaf leaf; Qdx *dx = self->qdx; int ret = 0; unsigned i = 0; int stop = QSTOP & self->flg; int skip = QSKIP & self->flg; Block *b = &leaf.b; Key *cmp = &self->key; if ( ! cmp->len ) { /* start at 1st leaf */ if ( b != GETLEAF(b, dx, 0, LOCK_READ) ) DONE((ERR_TRASH, "could not read leaf 0")); self->cur.len = 1; /* avoid QSAME match */ } else { /* locate key */ FindKey f; f.dx = dx; f.key = cmp; if ( FINDKEY(b, &f, LOCK_READ) ) DONE((ERR_TRASH, "could not find key")); i = (unsigned)(QENTMASK & f.e[0]); if ( !(QMATCH & f.e[0]) && QEQ == stop ) goto done; self->cur.len = 0; /* avoid QSAME match */ } if ( QUPTO <= stop ) cmp = &self->to; for ( ;; i=0 ) { /* blocks */ unsigned char *base = (unsigned char*)b; Dict *d; if ( ! b || b->lev ) return ERR_TRASH; d = b->dct + i; for ( ; ient; i++, d++ ) { unsigned short pos = LEAFPOS( d ); unsigned short vln = LEAFVLN( d ); unsigned short kln = d->kln; unsigned char *v = base+pos+kln; Ptr p; if ( skip ) { if ( kln == self->key.len && !memcmp(self->key.byt, base+pos, kln) ) continue; skip = 0; } if ( stop ) { int diff = memcmp( cmp->byt, base+pos, kln <= cmp->len ? kln : cmp->len ); switch ( stop ) { case QPF: if ( diff || kln < cmp->len ) goto moan; break; case QEQ: if ( diff || kln != cmp->len ) goto done; break; case QUPTO: if ( 0 >= diff ) goto done; break; case QINCL: if ( 0 > diff || (!diff && kln > cmp->len) ) goto done; break; default: /* unused */ moan: eRr(LOG_INFO, "loop ends on key '%b'(%d) %d %d", kln, base+pos, kln, diff, cmp->len ); goto done; } } /* if stop */ if ( !i && kln == self->cur.len && !memcmp(self->cur.byt, base+pos, kln) ) self->flg |= QSAME; else { self->flg &= ~QSAME; memcpy( self->cur.byt, base+pos, self->cur.len = kln ); } if ( self->qcb ) { self->vals = v; self->nvals = vln; if ( (ret = self->qcb(self)) ) goto done; } else { /* internal check */ Leaf ll; FindKey f; f.dx = dx; f.key = &self->cur; memcpy( self->cur.val.byt, v, self->cur.val.len = dx->vsz ); if ( FINDKEY(&ll.b, &f, LOCK_READ) ) DONE((ERR_TRASH, "could not find key")); if ( !(QMATCH & f.e[0]) || b->num != ll.b.num || i != (unsigned)(QENTMASK & f.e[0]) ) { eRr(ERR_TRASH, "OOPS! want %u.%u got %u.%u (match %4x)", b->num, i, ll.b.num, QENTMASK & f.e[0], f.e[0] ); qRdVal( &p, v, dx->vsz ); eRr(ERR_TRASH, "\twhen looking for '%b'/%u.%u.%u", self->cur.len, self->cur.byt, p.rid, p.tag, p.pos ); if ( ll.b.ent <= (QENTMASK & f.e[0]) ) eRr(ERR_TRASH, "\tI found nothing" ); else { Dict *dd = ll.b.dct + (QENTMASK & f.e[0]); unsigned char *bb = ((unsigned char*)&ll.b)+LEAFPOS(dd); qRdVal( &p, bb+dd->kln, dx->vsz ); eRr(ERR_TRASH, "\tI found '%b'/%u.%u.%u", dd->kln, bb, p.rid, p.tag, p.pos ); } } else eRr(LOG_TRACE, "key '%b' ok", self->cur.len, self->cur.byt ); } /* internal check */ } if ( ! b->nxt || !GETLEAF(b, dx, b->nxt, LOCK_READ) ) break; /* TODO: b now is the block that was our sibling when reading the old b. As long as we don't shift entries, there is no need to advance to next key that is actually greater !? */ } done: if ( ! self->qcb ) eRr(LOG_INFO, "checked %d keys in %u blks of %dK depth %u cmp/fget/lget %u/%u/%u", stat.key, dx->lln, dx->lsz>>10, dx->dpt, stat.cmp, stat.fget, stat.lget ); return ret; } /* qLoop */ int qInit ( Qdx *dx ) { Bloke blk; int size; /* defaults */ if ( !dx->ksz ) dx->ksz = 255; /* check leaves file */ size = fSize( dx->mqd ); if ( 0x3ff & size ) /* not on 1K boundary */ return eRr(LOG_ERROR, "mqd has bad size %dK + %d", size >> 10, (int)size & 0x3ff ); if ( size ) { /* read 1st leaf header */ Block b; int got = fPread( &dx->mqd, (unsigned char*)&b, sizeof(b), 0 ); if ( sizeof(b) != (unsigned)got ) return eRr(LOG_SYSERR, "could not get leaf head" ); if ( b.num ) return eRr(ERR_TRASH, "leaf has num %d", b.num ); if ( dx->typ && dx->typ != b.typ ) eRr(LOG_WARN, "typ want 0x02%2x got 0x02%2x", dx->typ, b.typ ); if ( QDX_COMPRS & dx->typ ) eRr(LOG_WARN, "sorry this version does not support compressed keys"); dx->typ = b.typ; dx->ksz = b.ksz; dx->ptr = b.ptr; } if ( QDX_LEAFPV & dx->typ ) dx->vsz = dx->ptr; else dx->vsz = (3&(dx->ptr>>6)) + 3+(3&(dx->ptr>>4)) + (7&dx->ptr); if (!size && dx->ksz > 255 - dx->vsz) dx->ksz = 255 - dx->vsz; eRr(LOG_VERBOSE, "dx ptr %2x vsz %d=%d+%d+%d", dx->ptr, dx->vsz, (3&(dx->ptr>>6)), 3+(3&(dx->ptr>>4)), (7&dx->ptr)); dx->lsz = 1<typ); if ( size % dx->lsz ) return eRr(ERR_TRASH, "size 0x%08x %d blksize", size, dx->lsz ); dx->lln = size / dx->lsz; if ( ! dx->lln && (!(DX_WRITE & dx->flg) || (newleaf(&blk.b, dx), wrtleaf(&blk.b, dx))) ) return eRr(LOG_SYSERR, "could not init new leaf file" ); #ifdef CPU_BIG_ENDIAN # define QDX_FORK (QDX_FORKBE | (env.psh-QDX_FORKSH)) #else # define QDX_FORK (QDX_FORKLE | (env.psh-QDX_FORKSH)) #endif size = fSize(dx->mqx.fil); if ( size ) { if ( size % env.psz ) /* not on env.psz boundary */ return eRr(LOG_ERROR, "mqx has bad size %dK + %d", size >> 10, (int)size & 0x3ff ); dx->fln = size >> env.psh; if ( (int)dx->fln != fMap(&dx->mqx, dx->fln) ) return eRr(LOG_ERROR, "could not map mqx"); dx->ftp = ((Block*)dx->mqx.map)->typ; if ( dx->ftp != QDX_FORK ) /* bad pagesize, endianess or flags */ return eRr(LOG_ERROR, "root has type %2x != %2x", dx->ftp, QDX_FORK ); dx->dpt = ((Block*)dx->mqx.map)->lev; } dx->fln = size / env.psz; if ( ! dx->fln ) { dx->ftp = QDX_FORK; eRr(LOG_INFO, "creating index type 0x%2x/0x%2x lsz %u psz %u ksz %u vsz %u", dx->typ, dx->ftp, dx->lsz, env.psz, dx->ksz, dx->vsz ); if ( 1 < dx->lln ) { if (ENV_EXCL!=env.wri || !(DX_WRITE & dx->flg)) return eRr(ERR_TRASH, "need excl write access to rebuild tree"); if ( qRemake(dx) ) return eRr(LOG_ERROR, "could not rebuild tree"); } else { /* init: link root to leaf */ Block *b = newfork(&blk.b, dx, 1); SETINT( ((unsigned char*)b)+(b->stk -= 4), 0 ); b->dct[0].pos = b->stk; /* 0, 0 */ b->ent = 1; if ( wrtfork(b, dx) ) return eRr(LOG_SYSERR, "\twhen writing root" ); fMap(&dx->mqx, 1); dx->dpt = 1; } } eRr(LOG_INFO, "psz %d mapped %d", env.psz, dx->mqx.npg ); return 0; } /* qInit */ void qFini ( Qdx *dx ) { fMap(&dx->mqx, 0); } /* qFini */ mlt/rdx.c0100644000076400001440000003667310175234160011457 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: rdx.c,v 1.1 2005/01/24 18:03:28 kripke Exp $ implementation of general db access functions. */ #include "core.h" /* type of rx posbytes in [4..8], lenbytes in [3..6], fldbytes in [0..2] */ #define RX_TYPE(posbytes,lenbytes,fldbytes) \ ((posbytes-4)<<4 | (lenbytes-3)<<2 | fldbytes) enum { /* by now, only the two 8 byte types are implemented */ RX_431 = RX_TYPE(4,3,1), RX_440 = RX_TYPE(4,4,0) }; /* ************************************************************ private types */ typedef union { lolo bar; char r[16]; } Xrf; /** header of first record index block */ typedef struct Rxh { char mag[3]; /* magic MRX or mrx */ char typ; unsigned mid; /* max record id == total number of used pointers */ /* for type of more than 8 byte: upper mid */ } Rxh; /* ************************************************************ data */ /* the magic we need depending on our endianess */ static char MAG_MRX[] = #ifdef CPU_BIG_ENDIAN "MRX" #else "mrx" #endif ; /* ************************************************************ private functions */ #define RLOCKSH(n) (void)FLOCKSH(dx->mrd, n) #define RLOCKEX(n) (void)FLOCKEX(dx->mrd, n) #define RLOCKUN(n) (void)FLOCKUN(dx->mrd, n) /* create a pointer from the least significant bytes of pos, len, fld buf must have dx->ptl bytes (up to 16 = 8+4+4) and the most strict alignment (i.e. 4 or 8) possible for dx->ptl returns buf supports only RX_431 and RX_440 */ static char *mkxrf ( char *buf, Rdx *dx, unsigned pos, unsigned len, unsigned fld ) { ((unsigned*)buf)[0] = pos; if ( RX_431 == dx->typ ) { if ( ~0xff & fld ) fld = 0; /* too big */ #ifdef CPU_BIG_ENDIAN /* the first = high order 3 bytes are len */ len = len<<8 | fld; #else /* the first = low order 3 bytes are len */ len = (0xffffff&len) | fld<<24; #endif } ((unsigned*)buf)[1] = len; return buf; } /* read pointer, return len if 0x0f00 & dx->typ, fld must not be 0, else *fld is untouched supports only RX_431 and RX_440 */ static unsigned rdxrf ( unsigned *pos, unsigned *fld, Rdx *dx, char *buf ) { unsigned len = ((unsigned*)buf)[1]; *pos = *(unsigned*)buf; if ( RX_431 == dx->typ ) { *fld = ((unsigned char *)buf)[7]; #ifdef CPU_BIG_ENDIAN len >>= 8; #else len &= 0xffffff; #endif } return len; } static void setXrf ( Rdx *dx, unsigned rid, unsigned pos, unsigned len, unsigned fld ) { Xrf xrf; unsigned off = rid*dx->ptl; eRr(LOG_INFO,"setXrf %d %d %d %d",rid,pos,len,fld); RLOCKEX(rid); if ( dx->mrx.npg ) { if ( (off+dx->ptl) > (dx->mrx.npg<ptl-1)>>env.psh); /* num pages we need */ if ( npg > dx->mrx.lim ) { /* at limit, drop the mapping completely */ eRr(LOG_WARN, "dropping mrx mapping at limit %u", dx->mrx.lim); npg = dx->mrx.lim = 0; } fMap(&dx->mrx, npg); } if ( (off+dx->ptl) <= (dx->mrx.npg<mrx.map + off, dx, pos, len, fld); if ( ENV_MSYNC & env.flg ) fMSync(&dx->mrx, off>>env.psh); goto done; } } fPwrite(&dx->mrx.fil, mkxrf(xrf.r, dx, pos, len, fld), dx->ptl, off); done: if ( rid > (unsigned)dx->mid ) { if ( dx->mrx.npg ) ((Rxh*)dx->mrx.map)->mid = rid; else fPwrite(&dx->mrx.fil, &rid, 4, 4); dx->mid = rid; } RLOCKUN(rid); } /* setXrf */ static unsigned getXrf ( unsigned *pos, unsigned *fld, Rdx *dx, int rid ) { unsigned ret; Xrf xrf; unsigned off = (unsigned)rid*dx->ptl; RLOCKSH(rid); if ( (off+dx->ptl) <= (dx->mrx.npg<mrx.map + off); else { #ifdef BUILD_SHMODE if ( ENV_SHARED==env.wri ) { /* might have been extended */ unsigned pgl; /* TODO: maybe we should guard this by the record lock ? */ if ( dx->mrx.npg ) dx->mid = ((Rxh*)dx->mrx.map)->mid; else fPread(&dx->mrx.fil, &dx->mid, 4, 4); if ( rid > dx->mid ) { ret = 0; goto out; } pgl = fSize(dx->mrx.fil) >> env.psh; if ( pgl != dx->mrx.npg ) fMap(&dx->mrx, pgl); if ( (off+dx->ptl) <= (dx->mrx.npg<mrx.map + off); goto out; } } #endif ret = rid > dx->mid || dx->ptl != fPread(&dx->mrx.fil, xrf.r, dx->ptl, off) ? 0 : rdxrf(pos, fld, dx, xrf.r); } #ifdef BUILD_SHMODE out: RLOCKUN(rid); #endif return ret; } /* getXrf */ static int rEmake ( Rdx *dx ) { char buf[65536+1]; /* need 64K buf for copying DO NOT SHRINK !!! */ int sz; char *p, *q; Xrf xrf; unsigned base = 0; /* of current block */ unsigned pos = 0; /* of last record */ /* record parsing starts at the first char of rid 1, so fld is initially 1 */ unsigned fld = 1; /* of last record */ unsigned nrid = 1; /* next rid = maxrid+1 */ unsigned xrid = 0; /* explicitly given */ char op = 0; int more; /* buf not empty flag */ char *last; /* of current block */ /* TODO: base on FBuf? */ if ( fTrunc(&dx->mrx.fil, 0) ) return eRr( LOG_ERROR, "could not trunc mrx" ); /* write signature */ memcpy( xrf.r, MAG_MRX, 3 ); xrf.r[3] = dx->typ; memset( xrf.r+4, 0, dx->ptl-4 ); fPwrite( &dx->mrx.fil, xrf.r, dx->ptl, 0 ); /* loop the masterfile */ fSeek( &dx->mrd, 0 ); last = (p = buf) + fRead( &dx->mrd, buf, 8192 ) - 1; more = last > buf; /* one byte is no byte ;) */ for (;;) { /* records */ unsigned len, rid; for (;;) { /* lines and stuff to end of record */ if ( p < last ) { /* have one lookahead */ if ( LF != *p++ ) continue; /* the tight loop ... or use memchr ? */ if ( LF != *p ) { /* now p <= last */ fld++; /* count field, unless we really recognize a opline */ if ( 'W' != *p ) continue; sz = last - p; /* avail after p */ if ( sz > 127 ) /* longer -> no opline */ sz = 127; if ( ! sz || ! (q = memchr(p+1, LF, sz)) ) { if ( sz >= 127 || ! more ) continue; /* too long or undelimited last */ p--; /* back to \n, so we come here again */ goto gimmemore; } if ( q < p+3 || p[2] < '0' || '9' < p[2] ) continue; /* TODO: take a closer look at whether the whole line makes sense */ if ( op ) { /* yeah, two metas in sequence! weird stuff! */ p--; /* step back to newline */ pos = base+(p-buf); /* fake pos as if we had no line at all */ break; /* go handle the PREVIOUS opline */ } op = *p; xrid = a2i( p+2, q-p-2 ); continue; } break; } gimmemore: LOG_DBG( LOG_DEBUG, "MORE %d at pos %d base %d p +%d last +%d", more, pos, base, p-buf, last-buf ); if ( !more ) goto schicht; /* german: done */ base += p - buf; /* shift out bytes before p */ len = last-p; /* bytes to keep after p; < 128 */ if ( len ) /* we're probing for more lookahead */ memmove( buf, p, 1+last-p ); else /* typically */ *buf = *p; /* but save the last dance */ p = buf; last = buf + len; /* reload */ if ( 0 < (sz = fRead( &dx->mrd, buf+1+len, 8192 )) ) { last += sz; continue; } more = 0; /* but yet, finish this up */ /* since *buf = *last was the files last character, we'd expect a newline */ if ( last == p ) p = buf+(LF==*buf ? 1 : 2); /* pretend buf started \n */ if ( ! len ) break; /* else try again opline */ } /* lines and stuff */ /* now p is on a delimiting blank lines \n -- or such ... */ len = base + (p-buf) - pos; /* >= 0 */ rid = xrid ? xrid : nrid; eRr( LOG_INFO, "mrx %c %d(%d/%d) pos %d len %d", op?op:'>', rid, xrid, nrid, pos, len ); if ( base + (p-buf) < pos ) /* FOO !!! */ len = 0; if ( rid && (len || op) ) fPwrite( &dx->mrx.fil, mkxrf( xrf.r, dx, pos, len, fld), dx->ptl, rid*dx->ptl ); else if (!more) break; pos = base + (p-buf) + 1; /* next starts after p */ if (!op) nrid++; /* continue after this */ else if (nrid <= xrid) nrid = xrid+1; xrid = fld = op = 0; } schicht: ; dx->mid = nrid-1; fPwrite(&dx->mrx.fil, &dx->mid, 4, 4); fTrunc(&dx->mrx.fil, ((1+dx->mid)*dx->ptl+env.psz-1)>>env.psh<fav < n || LAVL(l) < len+23 ) lExtend(l, len+23, n); /* the call alone is about 2% ! */ if ( 0 >= (got = fPread(&dx->mrd, rd = l->buf+22, len, pos)) ) return got; if ( len != (unsigned)got ) { if ( first ) return eRr(ERR_TRASH, "wanted %d bytes got %d at %d", len, got, pos); len = got; cont = 0; } e = (p = rd) + len; /* check header for rid@pos, mv p to leader */ if ( 2 < len && 'W'==*p && TAB==p[1] ) { /* else plain record */ int val; p += 2; if ( (got = a2il(p,e-p,&val)) ) { if ( val != rid ) return eRr(ERR_TRASH, "wanted rid %d got %d at %d", rid, val, pos); p += got; if ( p < e && '@'==*p ) { p++; if ( (got = a2il(p,e-p,&val)) ) { if ( mpos && pos >= mpos ) { if ( val < 0 || (unsigned)val >= pos ) return eRr(ERR_TRASH, "bad hpos %d at %d", val, pos); pos = (unsigned)val; if ( first ) len += len>>1; /* give some safety margin */ continue; } p += got; } } } } if ( mpos && pos >= mpos ) return 0; if ( !first ) { /* check for complete record */ for ( q=p; q>1); /* try using more space */ continue; ok: len = q+1 - rd; } LOG_DBG( LOG_TRACE, "'%.*s'", len, rd ); /* go for it */ f = l->fld + head; f->len = u2a(l->buf, rid); /* print header */ l->buf[f->len++] = '@'; f->len += u2a(l->buf+f->len, pos); /* now f->len <= 21 */ if ( p == rd ) { /* got no header -- rec data starts at rd */ if ( l->buf < (f->val = rd - f->len - 1) ) memmove(f->val, l->buf, f->len); /* just to keep it contiguous */ f->val[f->len] = LF; } else { /* TAB-leader-LF at p */ memmove(f->val = p - f->len, l->buf, f->len); q = memchr(p,LF,e-p); if ( !q ) return eRr(ERR_TRASH, "missing LF"); f->len = q - f->val; } p = f->val + f->len + 1; f->tag = -1; if ( head ) l->fld->tag--; /* count in total length */ l->fav--; f++; *e = LF; /* sentinel for memchr */ for ( n=0; pfav ) { lExtend( l, 0, 16 ); f = LEND(l); } /* p += a2il( p, q-p, &f->tag ); */ if ( (min = ('-' == *p)) ) p++; for ( ; 10>(d = b36val[(unsigned char)*p]); p++ ) t = 10*t + d; f->tag = min ? -t : t; if ( TAB == *p ) p++; f->len = q - (f->val = p); l->fld->tag--; l->fav--; f++; n++; } if ( head ) l->fld[head].tag = -1-n; l->buf = e; return n+1; } return 0; } /* rRead */ int rWrite ( Rdx *dx, const Fld *r, int rid, int opos, unsigned siz ) { char buf[128 + 65536]; LBlk *blk; unsigned pos = 0, rlen; char *p, *b; int intro = 1, ret; if ( !(dx->flg & DX_WRITE) ) return eRr(ERR_BADF, "dx is not writable"); rlen = RLEN(r); if ( rid && dx->mid >= rid ) { unsigned fld; getXrf( &pos, &fld, dx, rid ); if ( 0<=opos && pos != (unsigned)opos ) return eRr(ERR_AGAIN, "pos %d != opos %d", pos, opos); } RLOCKEX(0); if ( ! rid || rid == dx->mid+1 ) { rid = dx->mid+1; /* assign new rid; mid updated in setXrf */ if ( !rlen || !r->len ) /* no header */ intro = 0; } if ( !siz ) siz = rSiz( r ); siz += 13*rlen + 128; /* field, header, trailing LF */ if ( sizeof(buf) >= siz ) { blk = 0; b = buf; } else { blk = mBlkAlloc( siz ); /* hmm ... maybe would be better ... */ b = blk->byt; /* ... to serialize and write several 64K chunks */ } p = b; if ( intro ) { *p++ = 'W'; *p++ = TAB; p += u2a( p, rid ); if ( pos ) { *p++ = '@'; p += u2a( p, pos ); } if ( rlen && r->len ) { *p++ = TAB; memcpy(p, r->val, r->len); p += r->len; } *p++ = LF; } p += rSer( p, r ); dx->flg |= DX_MODIF; pos = dx->rdl; dx->rdl += p-b; ret = fPwrite(&dx->mrd, b, p-b, pos); if ( ret == p - b ) { setXrf(dx, (unsigned)rid, pos, p-b, rlen); ret = rid; } else if (0 <= ret) /* should not happen */ ret = ERR_IDIOT; if ( blk ) mBlkFree( blk ); RLOCKUN(0); return ret; } /* rWrite */ int rInit ( Rdx *dx ) { int remake = 1; dx->rdl = fSize( dx->mrd ); { /* reasons why we should remake */ Rxh head; int mxlen = fSize( dx->mrx.fil ); if ( 8 != fRead(&dx->mrx.fil,&head,8) ) eRr(LOG_WARN, "mrx too short"); else if ( memcmp(MAG_MRX,head.mag,3) ) /* FOO! */ eRr(LOG_WARN, "mrx has black magic '%.*s'", 3, head.mag); else if ( 0x80 & head.typ ) eRr(LOG_WARN, "mrx has bad type 0x%04x", head.typ); else if ( (dx->typ && dx->typ != head.typ) ) /* other type configured */ eRr(LOG_WARN, "mrx type 0x%02x != cfg 0x%02x", head.typ, dx->typ); else if ( mxlen < (int)(1+head.mid)*dx->ptl ) eRr(LOG_WARN, "mrx too short for len %d", head.mid); else if ( mxlen % env.psz ) eRr(LOG_WARN, "mrx len mismatch pagesize %d", env.psz); else if ( (DX_WRITE&dx->mrx.flg) && fTime(dx->mrx.fil) < fTime(dx->mrd) ) eRr(LOG_WARN, "mrx older than data"); /* ignore with RO */ else { dx->typ = head.typ; /* but further checked below */ dx->mid = head.mid; remake = 0; } } switch (dx->typ) { default: eRr(LOG_WARN, "mrx type 0x%02x unsupported", dx->typ); case 0: dx->typ = RX_431; remake = 1; case RX_431: case RX_440: break; } dx->ptl = 8; if ( remake ) { if (ENV_EXCL!=env.wri || !(DX_WRITE & dx->flg)) return eRr(ERR_TRASH, "need excl write access to rebuild .mrx"); rEmake(dx); } fMap(&dx->mrx, fSize(dx->mrx.fil)>>env.psh); eRr(LOG_INFO, "mapped %d*%d = %d", 1+dx->mid, dx->ptl, dx->mrx.npg); return 0; } /* rInit */ void rFini ( Rdx *dx ) { if ( DX_MODIF & dx->flg ) { if ( dx->mrx.npg ) { ((Rxh*)dx->mrx.map)->mid = dx->mid; fMap(&dx->mrx, 0); } else fPwrite(&dx->mrx.fil, &dx->mid, 4, 4); } } /* rFini */ mlt/run0100755000076400001440000000057210175234220011232 0ustar gabiusers#!/bin/sh # # run demo server on test/cds # # $Id: run,v 1.1 2005/01/24 18:04:00 kripke Exp $ if [ ! -f ../test/cds.mqd ]; then echo "loading cds index ..." ../bin/malete qload -vi cds -f../test/ -wa <../test/cds.mqt fi if [ x != "x$TEST" ]; then ../bin/malete server -vd cds -f../test/ -w < #include #include #include #include #include /* exit */ #include #include #include #include /* memcpy et al */ #include #include #include #else #define WIN32_LEAN_AND_MEAN #define NONAMELESSUNION /* #include -lws2_32 */ #include /* -lwsock32 "good" enough ? */ #define close closesocket #endif #include "tool.h" static void stdsrv ( file f ) { List in; FIL_DEFBUF(f); lInit(&in, 0); while ( fGetr(lClr(&in), &fb) ) { dispatch(in.fld); SEOR(env.out); } } /* srv */ static char buf[0x2000]; static int fill; static int crlf; static int wrotehead; static int broken; static unsigned sock; /* response socket */ enum { SOL, /* at start of line */ TAG, /* in tag (only digits seen in line) */ VAL /* in val (some non-digit seen) */ }; static int flushsock () { int got = send(sock, buf, fill, 0); if ( 0 < got ) { if ( !(fill -= got) ) return 0; memmove(buf, buf+got, fill); return 0; } fill = 0; return broken = eRr(ERR_NO, "could not send"); } static void sinksock (Sink *lo, int eor) { Fld *f = lo->lst.fld, *e = f + RLEN(f); if ( (unsigned)-1 == sock ) goto reset; if ( wrotehead ) f++; for ( ; f < e; f++ ) { int maxfill = sizeof(buf)-13-f->len, l, n; char *p, *v; if ( 0 > maxfill ) maxfill = 0; /* make sure we have enough room */ while ( fill > maxfill ) if ( flushsock() ) goto reset; p = buf+fill; if ( wrotehead ) { p += i2a(p, f->tag); *p++ = TAB; } else { if ( 10>b36val[(unsigned)*f->val] ) { *p++ = 'W'; *p++ = TAB; } wrotehead = 1; } v = f->val; l = f->len; /* write field value */ while ( l >= (n = buf+sizeof(buf) - p) ) { memcpy(p, v, n); fill = sizeof(buf); if ( flushsock() ) goto reset; p = buf+fill; } memcpy(p, v, l); p += l; *p++ = LF; fill = p - buf; } /* for */ if ( eor ) { /* add a \n and flush */ if ( sizeof(buf) == fill && flushsock() ) /* pathological */ goto reset; buf[fill++] = LF; while ( fill && !flushsock() ) ; } reset: lReset(&lo->lst); } /* sinksock */ /* serve one request on sock */ static int srv1 () { #define CR 13 int state = SOL, neg = 0; char *p, *e, *n; unsigned l; List in; Fld *f = 0; crlf = 0; lInit(&in, 0); wantmore: /* really a for, but we need multi-level continue anyway */ if ( 0 >= (fill = recv(sock, p = buf, sizeof(buf), 0)) ) return eRr(LOG_WARN, "got %d in recv", fill); LOG_DBG(LOG_DEBUG, "got %d bytes", fill); e = p + fill; havemore: switch (state) { case SOL: switch (*p) { case CR: if (e == p+1) goto wantmore; /* ignore */ if (LF != p[1]) break; /* weird crap */ p++; case LF: /* got blank line */ if ( e > ++p ) return eRr(ERR_INVAL, "got %d excess bytes", e-p); goto gotit; /* break 2 */ } LNEWF(&in, 0, 80); if ( !f && (CT_IS(D,*p) || '-' == *p) ) { eRr(LOG_WARN, "no message name"); LNEWF(&in, 0, 80); } f = LLAST(&in); state = TAG; if ( (neg = '-' == *p) && e == ++p ) goto wantmore; case TAG: while ( CT_IS(D,*p) ) { f->tag = f->tag*10 + b36val[(unsigned char)*p]; if ( e == ++p ) goto wantmore; /* continue 2 */ } if ( neg ) f->tag = -f->tag; LOG_DBG(LOG_DEBUG, "got tag %d", f->tag); state = VAL; if ( TAB == *p && e == ++p ) goto wantmore; case VAL: n = memchr(p, LF, e-p); if ( !n ) l = e-p; else if ( (l = n-p) && 13 == n[-1] ) { if ( crlf ) l--; else if ( f == in.fld ) { l--; crlf = 1; LOG_DBG(LOG_DEBUG, "detected crlf"); } } LAPP(&in, p, l); if ( n ) { state = SOL; if ( e > (p = n+1) ) goto havemore; } goto wantmore; } gotit: fill = 0; wrotehead = 0; broken = 0; env.out->off = 0; lClr(&env.out->lst); dispatch(in.fld); SEOR(env.out); return broken; } /* srv1 */ /* ************************************************************ */ int server () { Ses s; Fld opt = { 0, 0, 0 }; fd_set lst; /* listening socks */ fd_set all; unsigned fdlen = 0; #ifndef WIN32 unsigned kids = 0; #endif sInit(ses = &s); FD_ZERO(&lst); FD_ZERO(&all); if ( env.opt ) for ( opt.val = 0; vGet(&opt, env.opt->fld, "S"); ) switch (opt.tag) { static const int yes = !0; struct sockaddr_in si; #ifndef WIN32 struct sockaddr_un su; #endif struct sockaddr *sa; int sal; Fld info; case 'S': /* socket */ info = opt; /* for message */ #ifndef WIN32 if ( opt.len && '/'==*opt.val ) { /* unix socket */ if ( opt.len > sizeof(su.sun_path)-1 ) { eRr(ERR_INVAL, "unix socket name '%.*s' too long", opt.len, opt.val); return 1; } memset(&su, 0, sizeof(su)); memcpy(su.sun_path, opt.val, opt.len); su.sun_path[opt.len] = 0; su.sun_family = AF_UNIX; unlink(su.sun_path); sa = (struct sockaddr*)&su; sal = sizeof(su); sock = socket(PF_UNIX, SOCK_STREAM, 0); } else #else { WSADATA wsadata; memset(&wsadata, 0, sizeof(wsadata)); if ( WSAStartup(2, &wsadata) ) { /* funny enough, winsock 1 says ok :) */ eRr(ERR_INVAL, "could not get winsock"); return 1; } } #endif { /* TCP socket */ memset(&si, 0, sizeof(si)); si.sin_family = AF_INET; si.sin_addr.s_addr = INADDR_ANY; si.sin_port = htons(2042); if ( opt.len ) { /* standard */ Fld port = opt; char *end = memchr(opt.val, ':', opt.len); if ( end /* host:port */ || CT_D!=lat1ct[(unsigned char)*opt.val] /* hostname */ || memchr(opt.val, '.', opt.len) ) { if ( !end ) end = opt.val+opt.len; if ( 9 >= (end - opt.val) && !memcmp("localhost",opt.val,end - opt.val) ) si.sin_addr.s_addr = htonl(INADDR_LOOPBACK); else { /* parse dotted decimal */ } if ( (port.len = opt.len - (end - opt.val)) ) { port.len--; port.val = end+1; } } if ( port.len ) si.sin_port = htons(V2I(&port)); } else { info.val = "*:2042"; info.len = 6; } sa = (struct sockaddr *)&si; sal = sizeof(si); sock = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP/*6*/); } if ( setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes)) ) { eRr(ERR_NO, "setsockopt"); return 1; } if ( bind(sock, sa, sal) ) { eRr(ERR_NO, "could not bind to '%.*s'", info.len, info.val); return 1; } if ( listen(sock, 64) ) { eRr(ERR_NO, "listen failed"); return 1; } eRr(LOG_INFO, "listening on '%.*s'", info.len, info.val); FD_SET(sock, &lst); FD_SET(sock, &all); if ( fdlen < sock+1 ) fdlen = sock+1; } /* done with options */ if ( !fdlen ) { stdsrv(env.in); return 0; } env.out->snk = sinksock; for (;;) { fd_set sele = all; int got = select(fdlen, &sele, 0, 0, 0); if ( 0 >= got ) { eRr(ERR_NO, "select said %d", got); continue; } for ( sock=0; got--; ) { while (!FD_ISSET(sock, &sele)) sock++; if ( FD_ISSET(sock, &lst) ) { struct sockaddr peer; unsigned /*socklen_t is broken*/ plen = sizeof(peer); unsigned nsock = accept(sock, &peer, &plen); #ifndef WIN32 if ( ENV_EXCL != env.wri ) { /* shared or read only */ if ( fork() ) { close(nsock); kids++; continue; } sock = nsock; while ( !srv1() ) ; exit(0); } #endif FD_SET(nsock, &all); if ( fdlen < nsock+1 ) fdlen = nsock+1; continue; } /* else serve a request on this socket */ /* TODO: use the socket's session */ if ( srv1() ) { close(sock); FD_CLR(sock, &all); } } #ifndef WIN32 for ( ;kids && 0 < waitpid(-1, 0, WNOHANG); kids--) ; #endif } return 0; } /* server */ mlt/sys.c0100644000076400001440000005331510233236727011476 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: sys.c,v 1.2 2005/04/25 15:49:25 kripke Exp $ I/O support for the openisis library. */ #include #include #include #include #include /* PATH_MAX */ #include #ifdef WIN32 #define WIN32_LEAN_AND_MEAN #define NONAMELESSUNION # include # include #else /* have operating system */ # define __USE_UNIX98 /* # define _POSIX_MAPPED_FILES # define _POSIX_SYNCHRONIZED_IO */ # include /* or try unixio */ # ifndef O_SYNC # if defined( __FreeBSD__ ) # define O_SYNC O_FSYNC # else # define O_SYNC 0 # endif # endif # include # include /* flock */ #if defined(__linux__) && !defined(__USE_GNU) # define __USE_GNU /* weird trigger to declare MREMAP_MAYMOVE */ #endif # include # include # include #endif /* WIN32 */ #include /* gettimeofday */ #include /* localtime */ #include "core.h" /* ************************************************************ private data */ /* ************************************************************ private functions */ static void *memordie ( int size ) { void *p = malloc( size ); if ( p ) return p; exit( 71/*EX_OSERR*/ ); /* exit handlers MUST NOT alloc memory */ return 0; } /* ************************************************************ private functions */ /* ************************************************************ public data */ Env env; /* ************************************************************ public functions */ void *mAlloc ( int size ) { void *p = memordie( size ); memset(p, 0, size); return p; } /* mAlloc */ void mFree ( void *mem ) { if ( mem ) free( mem ); } /* mFree */ void *mDup ( const void *str, int sz ) { void *m = memordie( 0<=sz ? sz : (sz = strlen( str ) + 1) ); memcpy( m, str, sz ); return m; } /* mDup */ char *mDupz ( const char *str, int sz ) { char *m = memordie( sz+1 ); memcpy( m, str, sz ); m[sz] = 0; return m; } /* mDupz */ LBlk *mBlkAlloc ( int size ) { LBlk *b = memordie( (((char*)(&((LBlk*)0)->byt))-(char*)0) + size ); b->nxt = 0; b->siz = size; memset(b->byt, 0, size); return b; } /* mBlkAlloc */ List *mListAlloc ( const char *name ) { return lInit(memordie(sizeof(List)), name); } /* mListAlloc */ int tUpd ( lolo *tm ) { lolo o; #ifdef WIN32 struct _timeb tb; _ftime( &tb ); if ( !tm ) return tb.time; o = *tm; *tm = tb.time*LOLO(1000) + tb.millitm; #else struct timeval tv; gettimeofday( &tv, 0 ); if ( !tm ) return tv.tv_sec; o = *tm; *tm = tv.tv_sec*LOLO(1000) + tv.tv_usec/1000; #endif return (int)(*tm - o); } /* tUpd */ static int tLoc ( struct tm *t, lolo *tm ) { lolo x; time_t tt; if ( !tm || !*tm ) tUpd( tm ? tm : (tm = &x) ); tt = (time_t)(*tm / 1000); #ifdef WIN32 /* did I mention it's not threadsafe ? */ *t = *localtime( &tt ); #else localtime_r( &tt, t ); #endif return (int)(*tm % 1000); } /* tLoc */ /* logically belongs to uti.c; here so tGtf may consider inlining */ void u2a0 ( char *p, unsigned u, unsigned n ) { for ( ; n--; u/=10 ) p[n] = '0' + (u % 10); } /* u2a0 */ int tGtf ( char *buf, lolo *tm ) { struct tm t; int millis = tLoc( &t, tm ); u2a0(buf, 1900+t.tm_year, 1900+t.tm_year); u2a0(buf+4, 1+t.tm_mon, 2); u2a0(buf+6, t.tm_mday, 2); u2a0(buf+8, t.tm_hour, 2); u2a0(buf+10, t.tm_min, 2); u2a0(buf+12, t.tm_sec, 2); buf[14] = 0; return millis; } /* tGtf */ char *tGtfm ( char *buf, lolo *tm ) { u2a0(buf+14, tGtf(buf, tm), 3); buf[18] = 0; return buf; } /* tGtfm */ #if 0 /* unused */ void tSleep ( lolo tm ) { #ifdef WIN32 Sleep( tm ); #else struct timespec ts; ts.tv_sec = tm / 1000; ts.tv_nsec = 1000000 * (int)(tm % 1000); nanosleep( &ts, 0 ); #endif } /* tSleep */ #endif #ifndef WIN32 static const char *lmode (int f) { switch ( (FIL_BLOCK|FIL_TLOCK|FIL_WR) & f ) { case FIL_BLOCK|FIL_WR: return "wait ex"; case FIL_BLOCK: return "wait sh"; case FIL_TLOCK|FIL_WR: return "test ex"; case FIL_TLOCK: return "test sh"; } return "unlock"; } #endif int fOpen ( file *fil, const char *name, int flags ) { int faillvl = FIL_TRY&flags ? LOG_VERBOSE : ERR_NO; #ifndef WIN32 int fd = -1; int f = !(FIL_WR&flags) ? O_RDONLY : ((FIL_RD&flags) ? O_RDWR : O_WRONLY); *fil = FIL_NONE; if (!name) { /* tempfile */ const char tpl[] = "/tmp/maleteXXXXXX"; char nam[sizeof tpl]; memcpy(nam, tpl, sizeof tpl); fd = mkstemp(nam); if (0 > fd) return eRr(ERR_NO, "could not create tmpfile"); eRr(LOG_VERBOSE, "tempfile '%s'", nam); unlink(nam); *fil = fd; return 0; } if ( FIL_CREAT & flags && FIL_WR & flags ) f |= O_CREAT; if ( FIL_SYNC & flags ) f |= O_SYNC; if ( FIL_TRUNC & flags ) f |= O_TRUNC; fd = open( name, f, 00664 ); /* let umask finetune */ if ( 0 > fd ) return eRr( faillvl, "could not open '%s' %x", name, flags ); if ( FIL_FLOCK & flags ) { struct flock fl; memset(&fl, 0, sizeof(fl)); fl.l_type = FIL_WR&flags ? F_WRLCK : F_RDLCK; fl.l_whence = SEEK_SET; fl.l_start = 0; fl.l_len = 0; eRr(LOG_VERBOSE, "%s perm lock on '%s'", lmode(flags), name); if ( fcntl(fd, FIL_BLOCK&flags ? F_SETLKW : F_SETLK, &fl) ) { close(fd); return eRr(ERR_NO, "err %s perm lock on '%s'", lmode(flags), name); } } *fil = fd; #else HANDLE h; int acc = 0; int shr = FILE_SHARE_READ | (FIL_FLOCK&flags ? 0 : FILE_SHARE_WRITE); int cre = (FIL_CREAT & flags && FIL_WR & flags) ? (FIL_TRUNC & flags) ? CREATE_ALWAYS : OPEN_ALWAYS : (FIL_TRUNC & flags) ? TRUNCATE_EXISTING : OPEN_EXISTING; int f = (FIL_WR&flags) ? FILE_ATTRIBUTE_ARCHIVE : FILE_ATTRIBUTE_NORMAL; char nam[128]; *fil = FIL_NONE; if (!name) { /* tempfile */ GetTempPath(sizeof nam, nam); GetTempFileName(nam, "mtp", 0, nam); name = nam; eRr(LOG_VERBOSE, "tempfile '%s'", nam); acc = GENERIC_READ|GENERIC_WRITE; shr = 0; cre = TRUNCATE_EXISTING; f = FILE_ATTRIBUTE_TEMPORARY|FILE_FLAG_DELETE_ON_CLOSE; } else { if ( FIL_RD & flags ) acc |= GENERIC_READ; /* FILE_READ_DATA; */ if ( FIL_WR & flags ) acc |= GENERIC_WRITE; if ( FIL_SYNC & flags ) f |= FILE_FLAG_WRITE_THROUGH; } f |= FILE_FLAG_RANDOM_ACCESS; h = CreateFile( name, acc, shr, 0, cre, f, 0 ); if ( INVALID_HANDLE_VALUE == h ) return eRr( faillvl, "could not open '%s' %x", name, flags ); *fil = h; #endif return 0; } /* fOpen */ int fClose ( file *f ) { if ( FIL_NONE != *f ) { #ifndef WIN32 for ( fsync( *f ); close( *f ); errno = 0 ) if ( EINTR != errno ) { eRr( LOG_SYSERR, "could not close file %d", *f ); return errno ? -errno : -1; } #else CloseHandle( (HANDLE)*f ); #endif *f = FIL_NONE; } return 0; } /* fClose */ int fSize ( file f ) { #ifndef WIN32 { struct stat s; return fstat( f, &s ) ? 0 : s.st_size; } #else return GetFileSize( (HANDLE)f, 0 ); #endif } /* fSize */ unsigned fTime ( file f ) { #ifndef WIN32 struct stat s; return fstat( f, &s ) ? 0 : (unsigned)s.st_mtime; #else FILETIME foo; /* time since 160101011200 UTC in hundred nanoseconds !!! */ lulu bar; if ( ! GetFileTime( (HANDLE)f, 0, 0, &foo ) ) return 0; bar = ((lulu)foo.dwHighDateTime<<32 | (lulu)foo.dwLowDateTime) / LULU(10000000); return (unsigned)(bar - LULU(11644473600)); #endif } /* fTime */ int fRead ( file *f, void *buf, unsigned count ) { #ifndef WIN32 int got = read( *f, buf, count ); return 0 <= got ? got : eRr( ERR_NO, "could not read %d bytes from %d", count, *f ); #else DWORD got = 0; int ok = ReadFile( (HANDLE)*f, buf, count, &got, 0 ); return ok ? got : eRr( ERR_NO, "could not read %d bytes", count ); #endif } /* fRead */ int fWrite ( file *f, const void *buf, unsigned count ) { { #ifndef WIN32 int got = write( *f, buf, count ); return 0 <= got ? got : eRr( ERR_NO, "could not write %d bytes to %d", count, *f ); #else DWORD got = 0; int ok = WriteFile( (HANDLE)*f, buf, count, &got, 0 ); return ok ? got : eRr( ERR_NO, "could not write %d bytes", count ); #endif } } /* fWrite */ int fSeek ( file *f, unsigned offset ) { { #ifndef WIN32 int got = (int)lseek( *f, offset, SEEK_SET ); return (int)offset == got ? 0 : eRr( ERR_NO, "could not seek to %d", offset ); #else return 0xffffffff /* "INVALID_SET_FILE_POINTER" */ == SetFilePointer( (HANDLE)*f, offset, 0, FILE_BEGIN ) ? -1 : 0; #endif } } /* fSeek */ int fPread ( file *f, void *buf, unsigned count, unsigned offset ) { { #ifndef WIN32 int got = pread( *f, buf, count, offset ); return 0 <= got ? got : eRr( ERR_NO, "could not read %d bytes from %d at %d", count, *f, offset ); #else return 0xffffffff /* "INVALID_SET_FILE_POINTER" */ == SetFilePointer( (HANDLE)*f, offset, 0, FILE_BEGIN ) ? -1 : fRead( f, buf, count ); #endif } } /* fPread */ int fPwrite ( file *f, const void *buf, unsigned count, unsigned offset ) { { #ifndef WIN32 int got = pwrite( *f, buf, count, offset ); return 0 <= got ? got : eRr( ERR_NO, "could not write %d bytes to %d at %d", count, *f, offset ); #else return 0xffffffff /* "INVALID_SET_FILE_POINTER" */ == SetFilePointer( (HANDLE)*f, offset, 0, FILE_BEGIN ) ? -1 : fWrite( f, buf, count ); #endif } } /* fPwrite */ int fTrunc ( file *f, unsigned length ) { #ifndef WIN32 ftruncate( *f, length ); if ( length && (int)length != fSize(*f) ) { /* Solaris and FreeBSD guarantee that ftruncate can extend files, but Linux does not. */ static char nosh = 0; fPwrite(f, &nosh, 1, length-1); } #else /* SetFileValidData( (HANDLE)*f, length ); XP only */ return 0xffffffff /* "INVALID_SET_FILE_POINTER" */ == SetFilePointer( (HANDLE)*f, length, 0, FILE_BEGIN ) ? -1 : SetEndOfFile( (HANDLE)*f ) ? 0 : eRr( ERR_NO, "could not truncate %d", length ); #endif return 0; } /* fTrunc */ #ifdef BUILD_SHMODE int fLock ( file f, unsigned n, int flg ) { struct flock fl; memset( &fl, 0, sizeof(fl) ); fl.l_type = FIL_WR&flg ? F_WRLCK : flg ? F_RDLCK : F_UNLCK; fl.l_whence = SEEK_SET; fl.l_start = n; fl.l_len = 1; eRr( LOG_VERBOSE, "%s tmp lock %d", lmode(flg), n ); return fcntl(f, FIL_BLOCK&flg ? F_SETLKW : F_SETLK, &fl) ? eRr(ERR_NO, "err %s tmp lock %d", lmode(flg), n) : 0; } /* fLock */ #endif int fMOpen ( FMap *fm, const char *name, int flags ) { int ret; fm->map = 0; fm->npg = 0; if ( (ret = fOpen(&fm->fil, name, flags)) ) fm->flg = 0; else { #ifdef WIN32 /* save mapping name */ static const char pfx[] = "Malete"; int l = strlen(name); char *c; fm->nam = mAlloc(sizeof(pfx)+l); memcpy(fm->nam, pfx, sizeof(pfx)-1); memcpy(c = fm->nam+sizeof(pfx)-1, name, l+1); for (; *c; c++) if ( '\\' == *c ) *c = '/'; /* must not contain \ */ #endif fm->flg = flags; } return ret; } /* fMOpen */ int fMClose ( FMap *fm ) { if ( fm->map ) fMap(fm, 0); #ifdef WIN32 /* save mapping name */ if ( fm->nam ) { mFree(fm->nam); fm->nam = 0; } #endif return fClose(&fm->fil); } /* fMClose */ int fMap ( FMap *fm, unsigned npg ) { if ( npg > fm->lim && fm->npg == (npg = fm->lim) ) return npg; #ifndef WIN32 if ((FIL_WR & fm->flg) && npg && (((int)npg< fSize(fm->fil))) fTrunc(&fm->fil, npg<map ) { /* unmap or remap */ #ifdef __linux__ if ( npg ) { LOG_DBG(LOG_DEBUG, "remapping %d pages at %8x", fm->npg, fm->map); fm->map = mremap(fm->map, fm->npg<npg = 0; goto done; } #endif munmap(fm->map, fm->npg<map = 0; fm->npg = 0; if (!npg) return 0; /* get new map */ fm->map = mmap( 0, npg<flg) ? PROT_WRITE : 0), MAP_SHARED, fm->fil, 0 ); #ifdef __linux__ done: #endif LOG_DBG(LOG_DEBUG, "mapped %d pages at %8x", npg, fm->map); if ( MAP_FAILED != fm->map ) return (int)(fm->npg = npg); fm->map = 0; return eRr(ERR_NO, "mmap failed"); #else /* awfully complicated here -- need CreateFileMapping, MapViewOfFile stupid piece of shrott true mapping supported in NT family only, 9x copies to swap */ if ( fm->map ) { UnmapViewOfFile(fm->map); CloseHandle(fm->hdl); } fm->map = 0; fm->npg = 0; fm->hdl = 0; if (!npg) return 0; /* note that shared writing of a mapping does not work, because to be consistent, it needs to be the same mapping object, which we can not grow. However, windoze doesn't support shared writing anyway. */ { int size = fSize(fm->fil); if (FIL_WR&fm->flg || size > npg<>env.psh; if ( (fm->hdl = CreateFileMapping(fm->fil,0, (FIL_WR&fm->flg)?PAGE_READWRITE:PAGE_READONLY,0,size,fm->nam)) ) { if ( (fm->map = MapViewOfFile(fm->hdl, (FIL_WR&fm->flg)?FILE_MAP_WRITE:FILE_MAP_READ,0,0,size)) ) return fm->npg = npg; CloseHandle(fm->hdl); fm->hdl = 0; return eRr(ERR_NO, "MapViewOfFile %d failed", npg); } } return eRr(ERR_NO, "CreateFileMapping failed"); #endif } /* fMap */ int fMSync ( FMap *fm, unsigned page ) { char * buf = fm->map + (page<flg ? MS_SYNC : MS_ASYNC) ) ? eRr( ERR_NO, "msync" ) : 0; #else return FlushViewOfFile(buf, env.psz) ? 0 : eRr(ERR_NO, "msync"); #endif } /* fMSync */ int fSlurp ( char **buf, int sz, const char *name, int opt ) { int size; int ret; char *p = *buf; file f; if ( (ret = fOpen( &f, name, FIL_RD|(opt ? FIL_TRY : 0) )) ) return ret; size = fSize( f ); if ( 0 >= size ) return size; if ( size > sz ) { eRr(ERR_INVAL, "file '%.*s' too big: %d > %d", 30, name, size, sz); goto done; } if ( ! p && !(p = mAlloc(size)) ) { ret = ERR_NOMEM; goto done; } ret = fRead( &f, p, size ); if ( size == ret ) *buf = p; else { if ( ret >= 0 ) ret = eRr( ERR_IO, "OOPS! got %d of %d bytes from '%.*s'", ret, size, 30, name ); if ( ! *buf ) free( p ); } done: fClose( &f ); return ret; } /* fSlurp */ int fGets ( FBuf *b ) { char *c; int n; if ( b->l ) { b->p += b->l; b->o += b->l; b->l = 0; } if ( b->m ) { b->p++; b->o++; b->m--; b->n++; } for (;;) { /* now have b->m bytes at b->p+b->l */ if ( b->m ) { /* add all or part of this b->l */ if ( (c = memchr(b->p+b->l, LF, b->m)) ) { b->m -= c - b->p - b->l; b->l = c - b->p; return !0; } b->l += b->m; b->m = 0; if ( b->l == b->s ) /* buffer exhausted */ return !0; if ( b->p != b->b ) memmove(b->b, b->p, b->l); } b->p = b->b; if ( 0 >= (n = fRead(&b->f, b->p+b->l, b->s-b->l)) ) /* eof */ return b->l; /* but had line */ b->m = n; } } /* fGets */ int fGetr ( List *l, FBuf *fb ) { if ( !fGets(fb) ) return 0; while ( fb->l ) { int tag; int n = a2il(fb->p, fb->l, &tag); if ( !l->fld->tag ) { /* init rec */ if ( n ) /* this is a data field, add blank header */ LPREF(l, -1, 0); else /* use this one as header */ tag = -1; } if ( n < (int)fb->l && TAB == fb->p[n] ) n++; LADD(l, tag, fb->p+n, fb->l-n); if ( !fGets(fb) ) break; } return !0; } /* fGetr */ static void fSinkf (Sink *lo, int eor, int full) { Fld *f = lo->lst.fld, *e = f + RLEN(f); file to = (file)lo->dst; if ( FIL_NONE == to ) goto reset; for ( f++; f < e; f++ ) { char buf[8192]; int l; if ( full ) { l = i2a(buf, f->tag); buf[l++] = '\t'; } else { if ( !f->len ) continue; l = 0; } if ( f->len < sizeof(buf)-l ) { memcpy(buf+l, f->val, f->len); buf[l += f->len] = '\n'; fWrite(&to, buf, l+1); } else { if ( l ) fWrite(&to, buf, l); if ( f->len ) fWrite(&to, f->val, f->len); buf[0] = '\n'; fWrite(&to, buf, 1); } } if ( eor ) fWrite(&to, "\n", 1); reset: lReset(&lo->lst); } /* fSinkf */ void fSinkl (Sink *lo, int eor) { fSinkf(lo, eor, 0); } /* fSinkl */ void fSinkr (Sink *lo, int eor) { Fld *f = lo->lst.fld; file to = (file)lo->dst; char buf[8192]; int l; if ( FIL_NONE == to ) goto reset; if ( f->tag ) { /* not empty */ if ( !lo->off && f->len ) { /* skip header if off or empty */ if ( 10>b36val[(unsigned)*f->val] ) {/* starts with digit */ buf[0] = 'W'; buf[1] = TAB; l = 2; } else l = 0; if ( f->len < sizeof(buf)-l ) { memcpy(buf+l, f->val, f->len); buf[l += f->len] = '\n'; fWrite(&to, buf, l+1); } else { if ( l ) fWrite(&to, buf, l); if ( f->len ) fWrite(&to, f->val, f->len); fWrite(&to, "\n", 1); } } } fSinkf(lo, eor, 1); if ( ! eor ) { lo->off += f->tag; /* no of fields passed by */ return; } reset: lo->off = 0; lClr(&lo->lst); } /* fSinkr */ int eOut ( int tag, const char *fmt, ... ) { if ( env.out ) { va_list ap; va_start( ap, fmt ); lOut(&env.out->lst, tag, 0, fmt, ap); va_end( ap ); SINK(env.out); } return tag; } /* eOut */ int eRr ( int tag, const char *fmt, ... ) { va_list ap; if ( ERR_NO == tag ) #ifndef WIN32 switch ( errno ) { case EINTR: case EAGAIN: tag = ERR_AGAIN; break; case EFAULT: case ENAMETOOLONG: case ELOOP: tag = ERR_FAULT; break; /* structurally bad address */ case EBADF: case ENOENT: case ENOTDIR: tag = ERR_BADF; break; /* file "does not exist" */ case EIO: case EEXIST: case EISDIR: case EACCES: case ENXIO: case ENODEV: case EROFS: case ETXTBSY: case ENOSPC: case EPIPE: case ESPIPE: tag = ERR_IO; break; /* file "is not accessible" */ case ENOMEM: case EMFILE: case ENFILE: tag = ERR_NOMEM; break; case EBUSY: tag = ERR_BUSY; break; default: tag = ERR_INVAL; } #else tag = ERR_INVAL; #endif if ( env.log > tag ) return tag; if ( !env.err ) return tag; va_start( ap, fmt ); lOut(&env.err->lst, -tag, 0, fmt, ap); va_end( ap ); /* logging afterburner */ if ( ERR_NOMEM >= tag && tag >= ERR_AGAIN ) { char *syserr = #ifndef WIN32 errno ? strerror(errno) : 0; #else 0; char buf[256] = ""; if ( FormatMessage( FORMAT_MESSAGE_FROM_SYSTEM|FORMAT_MESSAGE_IGNORE_INSERTS, 0, GetLastError(), 0, buf, sizeof(buf)-1, 0 ) ) syserr = buf; buf[ sizeof(buf)-1 ] = 0; #endif if ( syserr ) { LAPP(&env.err->lst, ": ", 2); LAPPS(&env.err->lst, syserr); } } SINK(env.err); /* just trigger the sink, buffering should be handled there */ return tag; } /* eRr */ /* NOTE: no shared lib support */ void cFini () { dCloseAll(); /* close env's buffers */ if ( env.out ) env.out->snk(env.out, 1); if ( env.err ) env.err->snk(env.err, 1); } /* cFini */ void cInit ( List *opts, Sink *out, Sink *err ) { Sink **oe; Fld opt = { 0, 0, 0 }; int i; if ( env.out ) return; memset(&env, 0, sizeof(env)); env.opt = opts; #ifndef WIN32 env.in = 0; #else if ( INVALID_HANDLE_VALUE == (env.in = GetStdHandle(STD_INPUT_HANDLE)) ) env.in = FIL_NONE; #endif env.out = out; env.err = err; oe = (Sink**)&env.out; for ( i=0; i<2; i++ ) { oe = i ? &env.err : &env.out; if ( *oe ) continue; *oe = (Sink*)mAlloc(sizeof(Sink)); #ifndef WIN32 (*oe)->dst = (void*)(i + 1); #else if ( INVALID_HANDLE_VALUE == ((*oe)->dst = GetStdHandle(i?STD_ERROR_HANDLE:STD_OUTPUT_HANDLE)) ) fOpen((file*)&(*oe)->dst, i ? "err.txt" : "out.txt", FIL_WR|FIL_CREAT); #endif if ( i ) { lInit(&env.err->lst, "err"); env.err->snk = fSinkl; } else { lInit(&env.out->lst, 0); env.out->snk = fSinkr; } } /* check pagesize */ #ifndef WIN32 env.psz = sysconf(_SC_PAGESIZE); /* getpagesize(); */ #else { SYSTEM_INFO si; GetSystemInfo(&si); env.psz = si.dwPageSize; /* usually also 4k */ } #endif { int t = env.psz; env.psh = 0; while (t >>= 1) env.psh++; if ( env.psz != (unsigned)1< CPU_PAGE_SHIFT ) { eRr(ERR_TRASH, "weirdness! pagesize %d shift %d max %d", env.psz, env.psh, CPU_PAGE_SHIFT); exit(1); } if ( env.psh < 12 ) { /* but using multiples is ok */ eRr(LOG_WARN, "page_shift is %d !!!", env.psh); env.psz = 1<<(env.psh = 12); } } /* defaults */ env.log = LOG_WARN; env.wri = ENV_EXCL; env.qml = env.rml = 1<<(30-env.psh); /* map 1 GB at most */ if ( opts ) for ( opt.val = 0; vGet(&opt, opts->fld, "vw"); ) switch (opt.tag) { case 'v': /* logging: first char selects level, a 'b' as second sets buffering */ if ( opt.len ) { switch ( *opt.val ) { case '0': case 'o': env.log = 0; break; case '1': case 'f': env.log = LOG_FATAL; break; case '2': case 's': env.log = LOG_IOERR /*LOG_SYSERR*/; break; case '3': case 'e': env.log = LOG_ERROR; break; case '4': case 'w': env.log = LOG_WARN; break; case '5': case 'i': env.log = LOG_INFO; break; case '6': case 'v': env.log = LOG_VERBOSE; break; case '7': case 'd': env.log = LOG_DEBUG; break; case '8': case 't': env.log = LOG_TRACE; break; case '9': case 'a': env.log = LOG_ALL; break; } if ( 1 < opt.len && 'b' == opt.val[1] ) env.flg |= ENV_BUFFER; } break; case 'w': /* write/concurrency mode */ if ( opt.len ) switch ( *opt.val ) { case '0': env.wri = ENV_RO; break; case 's': #ifdef BUILD_SHMODE env.wri = ENV_SHARED; env.flg |= ENV_MSYNC; #else eRr(ERR_INVAL, "no shared mode build"); #endif break; } } /* if (in) for switch */ atexit( cFini ); } /* cInit */ mlt/tool.h0100644000076400001440000000256610240340252011627 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2004 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ #ifndef TOOL_H /* $Id: tool.h,v 1.3 2005/05/11 08:22:02 kripke Exp $ interface of the Malete tools */ #define TOOL_VERSION "1.0.2" #include "pw.h" extern int server (); #ifndef BUILD_NOCDS extern int cdsimp (Db *db, Fld *args); extern int cdsexp (Db *db, Fld *args); extern int cdsuse (); #endif #ifndef BUILD_NOIIF extern int iifimp (Db *db, Fld *args); extern int iifexp (Db *db, Fld *args); extern int iifuse (); #endif #define TOOL_H #endif /* TOOL_H */ mlt/uti.c0100644000076400001440000005256310175234306011461 0ustar gabiusers/* The Malete project - the Z39.2/Z39.50 database framework of OpenIsis. Version 0.9.x (patchlevel see file Version) Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA see README for more information EOH */ /* $Id: uti.c,v 1.1 2005/01/24 18:04:54 kripke Exp $ utilities */ #include /* strlen */ #include #include "core.h" /* ************************************************************ data */ const char b36dig[36] = { '0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f','g','h', 'i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z' }; const char b36val[256] = { 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,36,36,36,36,36,36, 36,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24, 25,26,27,28,29,30,31,32,33,34,35,36,36,36,36,36, 36,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24, 25,26,27,28,29,30,31,32,33,34,35,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36, 36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36 }; const unsigned char lat1up[256] = { 0,' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ', ' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ', ' ','!','"','#','$','%','&','\'','(',')','*','+',',','-','.','/', '0','1','2','3','4','5','6','7','8','9',':',';','<','=','>','?', '@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O', 'P','Q','R','S','T','U','V','W','X','Y','Z','[','\\',']','^','_', '`','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O', 'P','Q','R','S','T','U','V','W','X','Y','Z','{','|','}','~',' ', ' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ', ' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ', ' ',161,162,163,164,165,166,167,168,169,170,171,172,173,174,175, 176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191, 'A','A','A','A','A','A','A','C','E','E','E','E','I','I','I','I', 'T','N','O','O','O','O','O',215,'O','U','U','U','U','Y','T','S', 'A','A','A','A','A','A','A','C','E','E','E','E','I','I','I','I', 'T','N','O','O','O','O','O',247,'O','U','U','U','U','Y','T','Y' }; const unsigned char lat1ct[256] = { /* NUL STX SOT ETX EOT ENQ ACK BEL */ CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W, /* BS HT LF VT FF CR SO SI */ CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W, /* DLE DC1 DC2 DC3 DC4 NAK SYN ETB */ CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W, /* CAN EM SUB ESC FS GS RS US */ CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W,CT_W, /*blank ! " # $ % & ' */ CT_W,CT_S,CT_S,CT_S,CT_S,CT_S,CT_S,CT_S, /* ( ) * + , - . / */ CT_S,CT_S,CT_S,CT_S,CT_S,CT_S,CT_S,CT_S, /* 0 1 2 3 4 5 6 7 */ CT_D,CT_D,CT_D,CT_D,CT_D,CT_D,CT_D,CT_D, /* 8 9 : ; < = > ? */ CT_D,CT_D,CT_S,CT_S,CT_S,CT_S,CT_S,CT_S, /* @ A B C D E F G */ CT_S,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, /* H-O */ CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, /* P-W */ /* X Y Z [ \ ] ^ _ */ CT_A,CT_A,CT_A,CT_S,CT_S,CT_S,CT_S,CT_I, /* ` a b c d e f g */ CT_S,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A,CT_A, /* x y z { | } ~ DEL */ CT_A,CT_A,CT_A,CT_S,CT_S,CT_S,CT_S,CT_S, /* 32 C1 controls */ 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , /* 32 mostly symbols */ 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , /* 64 Latin alphas including 2 symbols */ CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, 0 , CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L, 0 , CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L,CT_L }; /* ************************************************************ general utilities */ int a2il ( const char *p, int l, int *res ) { const char *s = p; const char *e; char op = 0; int d, ret = 0; if ( 0 > l ) { /* 0 terminated */ while ( ' ' >= (unsigned char)*p ) if ( !*p++ ) return p - s -1; e = p+11; /* max useful, we bail out on 0 byte */ } else for ( e = p+l; p < e && ' ' >= (unsigned char)*p; ) /* skip white */ p++; if ( p < e ) switch ( *p ) { case '-': if ( p+1 == e ) ret = 1; /* so sole '-' is -1 */ case '~': /* sole ~ is ~0 */ case '+': op = *p++; } if ( p < e && '0' == *p && ++p < e && 'x' == *p ) while ( ++p < e && 16 > (d = b36val[(unsigned char)*p]) ) ret = (ret<<4) + d; else for ( ;p < e && 10 > (d = b36val[(unsigned char)*p]); p++ ) ret = 10*ret + d; if (0 != res) switch (op) { case '-': *res = -ret; break; case '~': *res = ~ret; break; default: *res = ret; } return p - s; } /* a2il */ int a2i ( const char *p, int l ) { int res; a2il (p, l, &res); return res; } int a2id ( const char *p, int l, int dflt ) { int res; if (0 > l) { l = (int) strlen (p); } if (l != a2il (p, l, &res)) { return dflt; } return res; } int i2a ( char *p, int i ) { if ( i ) { int min, dig = 0, j; if ( 0 < i ) min = 0; else { min = 1; *p++ = '-'; i = -i; } for ( j=i; j; j/=10 ) dig++; for ( *(p+=dig)=0; i; i/=10 ) *--p = '0' + (i % 10); return min+dig; } *p = '0'; return 1; } /* i2a */ int u2a ( char *p, unsigned u ) { if ( u ) { char *d = p+10; for ( ; u; u/=10 ) *--d = '0' + (u % 10); if ( p < d ) { char *e = p+10, *q = p; while ( d < e ) *q++ = *d++; return q-p; } return 10; } *p = '0'; return 1; } /* u2a */ /* ************************************************************ field functions */ int vGet ( Fld *dst, const Fld *src, const char *opt ) { char *p, *e = src->val + src->len; const char *q; if ( !dst->val || (p = dst->val+dst->len) < src->val || dst->val > e ) { /* initial */ p = src->val; if ( (!opt || '*' == *opt) && (!src->len || TAB == *p) ) { dst->tag = 0; dst->val = p; /* catch empty primary */ return 1 + (dst->len = 0); } } else if ( p >= e ) goto notfound; if ( p == src->val && (!opt || '*' == *opt) && TAB != *p ) dst->tag = 0; else { for (;;) { if ( e == p ) goto notfound; if ( TAB != *p++ ) continue; if ( e == p ) /* bad trailing tab */ goto notfound; /* p is on valid subfield indicator */ if ( !opt ) /* take any */ break; for (q=opt; *q; ) if (*p == *q++) goto takeit; /* break 2 */ } takeit: dst->tag = *p++; } dst->val = p; while ( e > p && TAB != *p ) p++; return 1 + (dst->len = p - dst->val); notfound: dst->val = 0; dst->len = 0; return 0; } /* vGet */ int vCmp ( const Fld *a, const Fld *b ) { unsigned l = a->len; int cmp; if ( l > b->len ) l = b->len; return (l && (cmp = memcmp(a->val, b->val, l))) ? cmp : (int)a->len - (int)b->len; } int vGt ( const Fld *a, const Fld *b ) { unsigned l = a->len; int cmp = 0; if ( l > b->len ) l = b->len; return (l && 0 < (cmp = memcmp(a->val, b->val, l))) ? 1 : !cmp && a->len > b->len; } /* ************************************************************ record functions */ unsigned rSiz ( const Fld *r ) { unsigned ret = 0; const Fld *e = r + RLEN(r); while ( r < e ) ret += (r++)->len; return ret; } /* rSiz */ const Fld *rGet ( const Fld *r, int tag, int *pos ) { const Fld *e = r + RLEN(r); const Fld *p = r + (pos ? *pos : 0); while ( ++p < e ) if ( tag == p->tag ) { if ( pos ) *pos = p-r; return p; } if ( pos ) *pos = e-r; return 0; } /* rGet */ const Fld *rKey ( const Fld *r, int tag, const char *key ) { const Fld *e = r + RLEN(r); const Fld *d = 0; unsigned kl = strlen(key); while ( ++r < e ) if ( tag == r->tag ) { if ( VKEY(r, key, kl) ) return r; if ( !r->len || TAB == *r->val ) d = r; } return d; } /* rKey */ const Fld *rDup ( const Fld *src, unsigned siz ) { int len = RLEN(src); const Fld *e = src + len; Fld *dst, *f; char *p; if ( !siz ) siz = rSiz( src ); f = dst = mAlloc( len*sizeof(Fld) + siz ); p = ((char*)dst) + len*sizeof(Fld); for ( ;src < e; src++, f++ ) { f->tag = src->tag; memcpy( f->val = p, src->val, f->len = src->len ); p += f->len; } return dst; } /* rDup */ int rSer ( char *buf, const Fld *f ) { #if 0 static const char repl[4] = { 0, VT, TAB, ' ' }; const char *s, *e; #endif const Fld *fe = REND(f); char *p = buf; while ( ++f < fe ) { p += i2a( p, f->tag ); *p++ = TAB; memcpy(p, f->val, f->len); p += f->len; #if 0 e = (s = f->val) + f->len; for (;;) { while ( stag -1; int i; if ( 2 > l ) return; /* fields to sort are r[1]..r[l] we use r[0]..r[l-1], since whs puts biggest at 0 in first step and sorts the others at 1..l-1 */ head = *r; *r = r[l]; /* weak-heap-forestify every node roots a weak heap AND is <= root (especially the leftist independent roots are <= root) */ for ( i=l; --i; ) { int g = i; /* granma is 1st left i or root, if i is totally leftist ;) */ while (!(1&g)) /* we're left child */ g >>= 1; /* shift granma right :( */ g >>= 1; r[i].len &= LENMSK; /* merge: swap if g < i */ if ( !gt(r+i, r+g) ) continue; tmp = r[i]; r[i] = r[g]; r[g] = tmp; /* swap */ r[i].len |= REVBIT; } /* re-weak-heap-forestify */ for ( i=l; --i>1; ) { int j=1, left; r[i].len &= LENMSK; /* i's REVBIT no longer needed */ /* follow path of left childs (independent roots) */ while ( i > (left = (j<<1) + ((REVBIT & r[j].len)?1:0)) ) j = left; /* put max of them and i (new root) at i */ do { /* merge: swap if i < j */ /* clear all REVBITs, as gt will probably access .len */ int rj = REVBIT & r[j].len; r[j].len &= LENMSK; /* now i,j both are clean */ if ( !gt(r+j, r+i) ) { r[j].len |= rj; continue; } tmp = r[i]; r[i] = r[j]; r[j] = tmp; /* swap */ /* toggle original REVBIT at j, which now is at i */ if ( ! rj ) r[j].len |= REVBIT; } while (j >>= 1); } r[1].len &= LENMSK; /* 2... have been cleared */ /* initial root still untouched biggest (never had REVBIT) */ r[l] = *r; *r = head; } /* rSort */ void rSortTag ( Fld *r ) /* could use radixsort here ... */ { Fld head, tmp; int l = -r->tag -1; int i; if ( 2 > l ) return; head = *r; *r = r[l]; for ( i=l; --i; ) { int g = i; while (!(1&g)) g >>= 1; g >>= 1; r[i].len &= LENMSK; if ( r[g].tag >= r[i].tag ) continue; tmp = r[i]; r[i] = r[g]; r[g] = tmp; /* swap */ r[i].len |= REVBIT; } for ( i=l; --i>1; ) { int j=1, left; while ( i > (left = (j<<1) + ((REVBIT & r[j].len)?1:0)) ) j = left; do { if ( r[i].tag >= r[j].tag ) continue; tmp = r[i]; r[i] = r[j]; r[j] = tmp; /* swap */ if ( REVBIT & r[i].len ) r[j].len &= LENMSK; else r[j].len |= REVBIT; } while (j >>= 1); } r[l] = *r; *r = head; while ( l-- ) /* clearing them later is significantly faster */ (++r)->len &= LENMSK; } /* rSortTag */ void rSortVal (Fld *r) { Fld head, tmp; int l = -r->tag -1; unsigned cl; /* compare length */ int i, cmp; if ( 2 > l ) return; head = *r; *r = r[l]; for ( i=l; --i; ) { int g = i; while (!(1&g)) g >>= 1; g >>= 1; if ( (cl = r[g].len) > r[i].len ) cl = r[i].len; if ( !cl ? !r[i].len : (0 < (cmp = memcmp(r[g].val,r[i].val,cl)) || (!cmp && r[i].len == cl)) ) /* i <= g: ok */ continue; tmp = r[i]; r[i] = r[g]; r[g] = tmp; r[i].len |= REVBIT; } for ( i=l; --i>1; ) { int j=1, left; r[i].len &= LENMSK; while ( i > (left = (j<<1) + ((REVBIT & r[j].len)?1:0)) ) j = left; do { unsigned clj = LENMSK & r[j].len; if ( (cl = r[i].len) > clj ) cl = clj; if ( !cl ? !r[j].len : (0 < (cmp = memcmp(r[i].val,r[j].val,cl)) || (!cmp && clj == cl)) ) /* j <= i: ok */ continue; tmp = r[i]; r[i] = r[j]; r[j] = tmp; if ( REVBIT & r[i].len ) { /* toggle j's REVBIT */ r[i].len &= LENMSK; r[j].len &= LENMSK; } else r[j].len |= REVBIT; } while (j >>= 1); } r[1].len &= LENMSK; r[l] = *r; *r = head; } /* rSortVal */ /* ************************************************************ list functions */ List *lInit ( List *l, const char *fmt, ... ) { memset( l, 0, sizeof(*l) ); l->blk = &l->bl0; l->fld = l->fl0; l->fav = DEFFIELDS; l->bok = l->buf = l->bl0.byt; l->end = l->buf + DEFBLKLEN; l->siz = 0; if ( fmt ) { /* print header */ va_list ap; va_start( ap, fmt ); lOut( l, -1, 0, fmt, ap ); va_end( ap ); } return l; } /* lInit */ #define LBLK_FREE( l ) \ while ( (l)->blk != &(l)->bl0 ) { \ LBlk *__nxt = (l)->blk->nxt; \ mBlkFree( (l)->blk ); \ (l)->blk = __nxt; \ } List *lClr ( List *l ) { l->fav -= l->fld->tag; l->fld->tag = 0; l->fld->val = l->bl0.byt; l->fld->len = 0; l->bok = l->buf = l->bl0.byt; l->end = l->buf + DEFBLKLEN; l->siz = 0; LBLK_FREE(l); return l; } /* lClr */ List *lReset ( List *l ) { if ( l->fld->val != l->bl0.byt ) { /* header has moved */ if ( l->fld->len > DEFBLKLEN ) l->fld->len = DEFBLKLEN; /* truncate header !!! */ memmove( l->bl0.byt, l->fld->val, l->fld->len ); l->fld->val = l->bl0.byt; } l->fav += -1 - l->fld->tag; l->fld->tag = -1; l->bok = l->buf = l->bl0.byt + l->fld->len; l->end = l->buf + DEFBLKLEN; l->siz = l->fld->len; LBLK_FREE(l); return l; } /* lReset */ void OPT_STDCALL lFini ( List *l ) { LBLK_FREE(l); if ( l->fld != l->fl0 ) mFldFree( l->fld ); } /* lFini */ int OPT_REGPARM lExtend ( List *l, unsigned need, int fields ) { LBlk *nju; Fld *contig; if ( 0 <= fields ) contig = 0; else { contig = LLAST(l); fields = -fields -1; need += contig->len; } LOG_DBG(LOG_VERBOSE, "lExtend need %u avl %d fields %u fav %d ctg %d", need, LAVL(l), fields, l->fav, contig); if ( l->fav < (unsigned)fields ) { int had = - l->fld->tag; Fld *f; if ( fields < had>>2 ) /* grow 25% */ fields = had>>2; else if ( fields < DEFFIELDS ) fields = DEFFIELDS; f = mFldAlloc(fields + had); memcpy( f, l->fld, had*sizeof(Fld) ); if ( l->fld != l->fl0 ) mFldFree( l->fld ); l->fld = f; l->fav = (unsigned)fields; } if ( LAVL(l) >= need ) return 1; l->siz += l->buf - l->blk->byt; /* add to secondary size */ nju = mBlkAlloc(need + DEFBLKLEN/2); nju->nxt = l->blk; l->blk = nju; l->buf = nju->byt; l->end = l->buf + nju->siz; /* >= requested */ if ( contig ) { memcpy( l->buf, contig->val, contig->len ); contig->val = l->buf; l->buf += contig->len; } return 1; } /* lExtend */ static const char at[] = { TAB, '@' }; int lArgv ( List *l, int tag, int argc, const char **argv ) { LNEWF( l, tag, 4000 ); if ( !argc ) return 0; if ( '-' != **argv ) { const char *a = *argv++; argc--; LAPPS( l, a ); } while ( argc-- ) { const char *a = *argv++; unsigned len = 1; if ( '-' == *a ) a++; else len = 2; LAPP( l, at, len ); LAPPS( l, a ); } return LLAST(l)->len; } /* lArgv */ List *lVar ( List *l, int argc, const char **argv ) { static const char tab = TAB; LNEWF( l, -1, 4000 ); if ( !argc ) return l; if ( '-' != **argv ) { const char *a = *argv++; argc--; LAPPS( l, a ); } while ( argc-- ) { const char *a = *argv++; if ( '-' == *a ) { a++; LAPP( l, &tab, 1 ); } else { LNEWF( l, 1, 100 ); } LAPPS( l, a ); } return l; } /* lVar */ int lCpy ( List *l, const Fld *src, unsigned need ) { int len = RLEN(src); const Fld *e = src + len; if ( !need ) need = rSiz( src ); if ( !lExtend(l, need, len) ) return 0; for ( ;src < e; src++ ) LADDF( l, src ); return len; } /* lCpy */ int lParse (List *l, const char *p, int s) { int state = LPS_VAL&s, bits = (LPS_NEG|LPS_CR)&s; const char *e = p + (LPS_LEN&s), *lf; unsigned len; Fld *f = LLAST(l); for (;;) switch (state) { case LPS_SOR: case LPS_SOL: switch (*p) { case CR: if (e == p+1) goto done; /* ignore */ if (LF != p[1]) break; /* weird crap */ p++; case LF: /* got blank line */ return e - p - 1; } LNEWF(l, 0, 80); if ( (LPS_SOR==state) && (CT_IS(D,*p) || '-' == *p) ) { eRr(LOG_WARN, "no message name"); LNEWF(l, 0, 80); } f = LLAST(l); state = LPS_TAG; if ( '-' != *p ) bits &= ~LPS_NEG; else { bits |= LPS_NEG; if ( e == ++p ) goto done; } case LPS_TAG: while ( CT_IS(D,*p) ) { f->tag = f->tag*10 + b36val[(unsigned char)*p]; if ( e == ++p ) goto done; } if ( LPS_NEG & bits ) f->tag = -f->tag; LOG_DBG(LOG_DEBUG, "got tag %d", f->tag); state = LPS_VAL; if ( TAB == *p && e == ++p ) goto done; case LPS_VAL: lf = memchr(p, LF, e-p); if ( !lf ) len = e-p; else if ( (len = lf-p) && CR == lf[-1] ) { if ( LPS_CR & bits ) len--; else if ( f == l->fld ) { len--; bits |= LPS_CR; LOG_DBG(LOG_DEBUG, "detected crlf"); } } LAPP(l, p, len); if ( lf ) { state = LPS_SOL; /* if ( f == l->fld ) STRIP W\t ? */ if ( e > (p = lf+1) ) continue; } goto done; } /* for switch */ done: return state|bits; } /* lParse */ int lOut ( List *l, int tag, const char *fmt, ... ) { va_list ap; int indir = !fmt; int bav; int len=0, i; unsigned u; char *p, *q=0, c=0; const char *tok = 0; va_start( ap, fmt ); if ( indir ) { va_list ap2; fmt = va_arg(ap, const char*); ap2 = va_arg(ap, va_list); va_end( ap ); ap = ap2; } LNEWF( l, tag, 4000 ); bav = LAVL(l); p = l->buf; for (;;) { if ( !*fmt ) goto extend; if ('%' != *fmt) { tok = fmt+1; while (*tok && '%'!=*tok) tok++; if ( bav < (len = tok-fmt) ) goto extend; memcpy(p, fmt, len); p += len; bav -= len; if ( !*(fmt = tok) ) goto extend; } /* now '%' == *fmt */ switch (c = *++fmt) { case 0: /* bah -- trailing % */ goto extend; case '0': /* only in %0nx */ if ( fmt[1] < '1' || '8' < fmt[1] || 'x' != fmt[2] ) continue; c = fmt[1]; tok = fmt + 2; case '2': /* meaning %02x */ case '4': case '8': if ( 'x' == fmt[1] ) tok = fmt + 1; /* optional */ if (0) { case 'x': c = '4'; case 'd': case 'u': tok = fmt; } if ( bav < (len = 12) ) goto extend; u = va_arg(ap, unsigned); switch (c) { case 'd': len = i2a(p, (int)u); break; case 'u': len = u2a(p, u); break; default: i = len = c - '0'; while (i--) { p[i] = b36dig[u & 0xf]; u >>= 4; } } fmt = tok; break; /* the numbers */ case 'c': if ( bav < (len = 1) ) goto extend; *p = (char)va_arg(ap, int); break; default: case '%': if ( bav < (len = 1) ) goto extend; *p = c; break; case 'b': /* bytes -- eats 2 args len, q */ len = 2*va_arg(ap, int); q = va_arg(ap, char*); if ( bav < len ) goto extend; the_bytes_thing: for (i=0; i>4]; p[i++] = b36dig[u & 0xf]; } break; case '.': /* only used as .*s */ if ( '*' != *++fmt || 's' != *++fmt ) continue; /* prolly bad ... */ case '*': /* shorthand */ len = va_arg(ap, int); case 's': q = va_arg(ap, char*); if ( 's' == c ) len = strlen(q); if ( 0 ) { Fld *v; case 'v': v = va_arg(ap, Fld*); len = v->len; q = v->val; } if ( bav < len ) goto extend; the_string_thing: /* for eivind aarset */ /* since we need to shift args to check len and cannot unshift */ memcpy(p, q, len); } p += len; bav -= len; fmt++; continue; extend: LLAST( l )->len += p - l->buf; l->buf = p; if ( !*fmt ) break; /* need len */ lExtend(l, len, -1); bav = LAVL(l); p = l->buf; switch (c) { case '.': case '*': case 's': goto the_string_thing; case 'b': goto the_bytes_thing; } fmt--; /* simply start over */ } if ( !indir ) /* else it's not ours and caller should va_end */ va_end( ap ); return len; } /* lOut */ php/Isis.php0100644000076400001440000000227410026051022012104 0ustar gabiusers php/Isis/Db.php0100644000076400001440000000652510233546216012452 0ustar gabiusers number hash (where name should be an "identifier") may contain some additional entries using non-identifier chars ... */ var $fdt; var $name; var $srv; function Isis_Db ( $fdt = null, $name = null, $server = null ) { $this->fdt = $fdt; $this->name = $name; $this->srv = $server; } /* internal helper to construct and send a request set $this->srv->dbg to true to get full listing of what's going on */ function req ( $type, $arg, $emb = null, $lst = null ) { $req = new Isis_Rec(); $req->head = $this->name.'.'.$type; if ( $arg ) $req->head .= "\t$arg"; if ( $emb ) foreach ($emb as $r) $req->embed($r); if ( $lst ) foreach ($lst as $l) $req->append(0,$l); return $this->srv->request( $req ); } /* query with $expr = null, fetch more results from previous query with $recs, fetch an array of records, else of mfns */ function query ( $expr = null, $recs = true, $head = null ) { if ( $expr && $recs && false === strpos($expr, '?') ) $expr .= '?'; // force fetch records $ret = $this->req('Q', $expr); $head = $ret->head; return $recs ? $ret->recs($this) : $ret->get(0); } /* read one or an array of mfns return one or an array of records */ function read ( $mfn ) { if ( is_array($mfn) ) { $ret = $this->req('R', null, null, $mfn); return $ret->recs($this); } $ret = $this->req('R', null, null, array($mfn)); $recs = $ret->recs($this); return $recs[0]; } /* read array of terms return one or an array of records */ function terms ( $from, $to = null ) { if (!is_null($to)) $from .= "\t$to"; $ret = $this->req('T', $from); return $ret->get(0); } /* write one or an array of records return an array of mfns written */ function write ( $rec ) { if ( !is_array($rec) ) $rec = array($rec); $ret = $this->req('W', null, $rec); return $ret->get(0); } /* unlike the other methods, this expects $req to be a prepared X request however, $name.X is prepended return res->head, which should be a comment */ function index ( $req ) { $pfx = $this->name.'.X'; if ( $req->head ) $req->head = $pfx."\t".$req->head; else $req->head = $pfx; $res = $this->srv->request( $req ); return $res->head; } } // class Isis_Db ?> php/Isis/Http.php0100644000076400001440000001276610142161651013043 0ustar gabiuserslen()) for ( $res = '', $t = reset($rec->tag), $v = reset($rec->val); ; $t = next($rec->tag), $v = next($rec->val) ) { if ( $nuts ) // use v42[]: the [] so we can send multiple values, // and the v cause numerical named params as [] // tend to crash PHP; GET always, POST sometimes works ... // (it's such a feeble ugly pile of shirt) $res .= 'v' . $t . '%5B%5D=' . urlencode($v); else $res .= $t . '=' . urlencode($v); if ( !--$i ) break; $res .= '&'; } return $res; } /** static we ARE on the nutty side -- try to turn http request into something usable see toUrl nuts above for the format UNFORTUNATELY we have no way to tell the original order of parameters. hmm ... for get we could resort to parse QUERY_STRING manually @param mixed $db_or_fdt a Isis_Db object or just it's fdt array to look up names. */ function fromReq ( $db_or_fdt = null, $server_map = null ) { if ( is_object($db_or_fdt) ) { $req = new Isis_Rec( '-db', $db_or_fdt ); $fdt = $db_or_fdt->fdt; } else { $req = new Isis_Rec; $fdt = $db_or_fdt; // array or null } if ( $server_map ) foreach ($server_map as $k => $v) if (isset($_SERVER[$k])) $req->append( $v, $_SERVER[$k] ); foreach (array($_GET,$_POST) as $in) foreach ($in as $k => $v) { if ( $fdt ) { if ( !is_int($k = $fdt[$k]) ) continue; } elseif ( is_numeric($k) ) $k = (int)$k; elseif ( sscanf( $k, 'v%d', &$i ) ) $k = (int)$i; else continue; if (is_string($v)) $req->append($k, $v); else foreach ($v as $w) // Wolfsburg $req->append($k, $w); } return $req; } /** pers 1: use pfsockopen and keep-alive pers -1: only keep-alive (for multiple requests within one run) pers 0: neither */ function Isis_Http ( $host, $url, $php = 0, $port = 80, $pers = 0 ) { $this->host = $host; $this->port = $port; $this->pers = 0 < $pers; $this->head = 'POST '.$url." HTTP/1.1\r\n" . 'Host: '.$host."\r\n" . "User-Agent: Isis\r\n" // try keep-alive. fnord does not support that with cgi . ($pers ? "Connection: keep-alive\r\n" : '') . "Content-Type: application/x-www-form-urlencoded\r\n" . 'Content-Length: '; $this->nuts = $php; $this->open(); } function request ( $req ) { if ( !$this->sock && !$this->open() ) return null; $url = Isis_Http::toUrl( $req, $this->nuts ); fwrite($this->sock, $this->head . strlen($url) ."\r\n\r\n" . $url ); // as opposed to folklore (http://www.php.net/manual/en/ref.stream.php), // there is NO trailing CRLF in 1.1 (RTFM err ... RFC 2068 4.1 Page 30) fflush($this->sock); $ka = 0; // keepalive -- default for 1.1, explicit for 1.0 $ok = ($line = fgets($this->sock)) && sscanf( $line, "HTTP/1.%d 200 OK", &$ka ); $chunk = 0; while ( ($line = fgets($this->sock)) && trim($line) ) { // echo "---$line---"; if ( !strncasecmp('connection: keep-alive', $line, 22) ) $ka = $ok; elseif ( !strncasecmp('connection: close', $line, 17) ) $ka = 0; elseif ( !strncasecmp('transfer-encoding: chunked', $line, 26) ) $chunk = 1; } if ( !$ok ) $res = null; else { $txt = ''; if ($chunk) { // RFC 2068 3.6 Page 25, 19.4.6 Page 154 while ( ($line = fgets($this->sock)) && trim($line) && sscanf( $line, "%x", &$chunk ) && $chunk ) { // error_log( "///chunk size $chunk///\n" ); $data = fread( $this->sock, $chunk ); $txt .= $data; fread( $this->sock, 2 ); // a CRLF } while ( ($line = fgets($this->sock)) && trim($line) ) // eat footer ; // echo "???$line???\n"; } else // HERE we rely on our other end being cooperative, // nicely ending response with blank line while ( ($line = fgets($this->sock)) && trim($line) ) { // echo "+++$line+++"; $txt .= $line; } $res = new Isis_Rec; $res->parse($txt, ISIS_REC_TEXT); } if ( !$ka ) { fclose( $this->sock ); $this->sock = 0; } return $res; } } // class Isis_Http ?> php/Isis/Rec.php0100644000076400001440000004270110142161673012631 0ustar gabiusers]*)=[^>]+>/', # dump substitutions '/>< pairs '/[><]/', # nuke other >< '/^\^./', # kill initial subfield spec '/\^a/', # ^a -> ; '/\^[b-i]/', # ^[b-i] -> , '/\^./' # others -> . ); $GLOBALS['ISIS_REC_MHL_REPL'] = array( '\1>', '; ', '', '', '; ', ', ', '. ' ); /** An ISIS(/IIF/Z39.2/ISO2709)-style record in pure PHP. This is only loosely connected to an Isis Database, most functions can be used without having a DB. Note: Most getter methods work regardless of the type of tags. However, the setters are strongly biased towards numeric tags. @version $Revision: 1.2 $ @license LGPL @package Isis */ class Isis_Rec { /** The associated database, if any. Better be an Isis_Db. @var object $db */ var $db = 0; /** The record's MFN ("master file number", a.k.a. rowid). A value of 0 means the record did not yet have a number in it's db, a write will be handled as insert. @var int $mfn */ var $mfn = 0; /** The record's header. For data records, this is mfn[@pos][\tleader], where leader can any be any MARC leader data. */ var $head = ''; /** Array of tags for the record. Keys and values better be integers (others might be ignored or yield unexpected results in some contexts). The count of the tag array is taken as length of the record. The tag array may have holes (unassigned ints) if unset is used. @var array $tag */ var $tag; /** Array of values for the record. Values better be strings (i.e. will be forced to strings by some operations). @var array $tag */ var $val; /** last tag accessed using function v */ var $v = null; // //////////////////////////////////////////////////////////// // static // /** static function to format a value. maybe used as Isis_Rec::fmt. @param string $val a value to be formatted @param string $fmt a format specification, defaults to 'MHL' - if $fmt is null, the value is returned unchanged - if $fmt starts with '&' or '%', that character is stripped and htmlspecialchars or urlencode, resp., is applied to the (each) value as last step - if $fmt starts with 'MHL' (the constant ISIS_REC_MHL), that is stripped and the classical MHL ISIS formatting applied to values (before & or % mangling, rarely needed with subfields) - if $fmt is (now) empty, the complete value is used - else we're going for subfields: - if $fmt starts with a hat, the hat is stripped an used as subfield delimiter (TAB otherwise) - if $fmt matches /^([^(]*)\((\d*)(\.\.(\d*))?/, the part from the first '(' on is stripped as occurence selector (note that an optional closing ')' and additional chars are ignored) - the (remaining) characters in $fmt are subfield names, '*' selects any subfield (including the initial unnamed, even if it's empty !), '' any without stripping subfield names - if there is a occurence or range selected, for every character in the $fmt the specified occurences are used (counted from 0). If either bound is empty, 0 is used. If a range is specified (.. given), an upper bound of 0 means up to end. By default, only the first occurence (0) is used. @return if there is only a single character (remaining) in $fmt, and only a single occurence selectedi (no ..), a string is returned. Else you've been asking for an array, and so an array is returned even if it contains only a single value. If more than one subfield name was specified or the '*', the names are used ('' for the initial). If a range is selected, the index is used (in addition). Fields are added by first looping over subfield names, then occurences. PHP may or may not loop the array in that order. Example: '^ab' gives keys 'a' and 'b', 'b(1..' gives keys '1','2'...m '^cab(..', gives 'c0', 'c1', ... 'a0', ... */ function fmt ( $val, $fmt = ISIS_REC_MHL ) { global $ISIS_REC_MHL_PAT, $ISIS_REC_MHL_REPL; if ( null == $fmt ) return $val; if ( '&' != $fmt{0} && '%' != $fmt{0} ) $mode = ''; else { $mode = '&' == $fmt{0} ? 'htmlspecialchars' : 'urlencode'; $fmt = substr($fmt,1); } if ($mhl = ('M' == $fmt{0} && ISIS_REC_MHL == substr($fmt,0,3)) ) $fmt = substr($fmt,3); if ( '' == $fmt ) { if ( $mhl ) $val = preg_replace($ISIS_REC_MHL_PAT, $ISIS_REC_MHL_REPL, $val); return $mode ? $mode($val) : $val; } if ( '^' != $fmt{0} ) $sep = "\t"; else { $sep = '^'; $fmt = substr($fmt,1); } $sub = explode($sep,$val); // this really is not for performance ... $n = count($sub); if ( false !== strpos($fmt,'(') // quick check && preg_match( '/^([^(]*)\((\d*)(\.\.(\d*))?/', $fmt, $m ) // actually matches everything with a ( in it ) { $fmt = $m[1]; // initial part if (!($fst = $m[2]) ) $fst = 0; if (!($range = $m[3])) $lst = $fst; elseif (!($lst = $m[4])) $lst = $n; } else $fst = $range = 0; $strip = $l = strlen($fmt); if (!$strip) { // plain mode $fmt = '*'; $l = 1; } if ( !($single = 1 == $l && !$range) ) $ret = array(); // echo "sub '$fmt'($fst..$lst) range $range single $single strip $strip n $n\n"; for ($i = 0; $i < $l; $i++ ) { $s = $fmt{$i}; $sel = '*' == $s ? 0 : 1; $o = -1; for ($j = $sel; $j < $n; $j++ ) { // echo "test '$s'($fst..$lst) $o\n"; if ( ($sel && $s != $sub[$j]{0}) || ++$o < $fst ) continue; if ( $lst < $o ) break; // ok, it's in range $v = ($strip && $j) ? substr($sub[$j],1) : $sub[$j]; if ( $mhl ) $v = preg_replace($ISIS_REC_MHL_PAT, $ISIS_REC_MHL_REPL, $v); if ( $mode ) $v = $mode($v); if ( $single ) return $v; $key = $sel ? (1 == $l ? '' : $s) : $strip && $j ? $sub[$j]{0} : ''; if ($range) $key .= $o; $ret[$key] = $v; if ( !$range ) break; } } return $ret; } // //////////////////////////////////////////////////////////// // ctor // /** create an ISIS record. @return object Isis_Rec a new ISIS record */ function Isis_Rec () { $this->tag = array(); $this->val = array(); if ( func_num_args() ) $this->add( func_get_args() ); } // //////////////////////////////////////////////////////////// // getters // /** @return the number of fields */ function len () { return count($this->tag); } /** try to look up non-numeric tags in the fdt */ function fdt ( &$tag ) { if ( !is_int($tag) ) { if (is_numeric($tag)) $tag = (int)$tag; else if ($this->db && $this->db->fdt && is_int($this->db->fdt[$tag])) $tag = $this->db->fdt[$tag]; } } /** reset tag and val array pointers */ function res () { reset($this->tag); reset($this->val); } /** get all values for tag as array */ function get ( $tag ) { $ret = array(); $this->fdt($tag); reset($this->tag); while (list($p,$t) = each($this->tag)) if ( $tag == $t ) $ret[] = $this->val[$p]; return $ret; } /** v is for value -- get the value of the next occurence of tag in the record. The position is reset by res or when using v with a different tag. @param mixed $fmt format to apply - if $fmt is null (or the value is null), the value is returned unmodified - if $fmt is a string, the value is formatted by Isis_Rec::fmt (returns string or array). - other values of $fmt are reserved for future extensions and currently return null. */ function v ( $tag = null, $fmt = null ) { if ( !is_int($tag) ) $this->fdt($tag); if ( $this->v != $tag ) { reset($this->tag); // no each on val needed $this->v = $tag; } $v = null; if ( is_null($tag) ) { if (list($p,$t) = each($this->tag)) $v = $this->val[$p]; } else while (list($p,$t) = each($this->tag)) if ( $tag == $t ) { $v = $this->val[$p]; break; } if (is_null($v) || is_null($fmt)) return $v; if ( is_string($fmt) ) return Isis_Rec::fmt($v,$fmt); return null; } /** return an array of subrecords */ function recs ( $db = null ) { $ret = array(); $n = 0; reset($this->tag); while (list($p,$t) = each($this->tag)) if ( 0 > $t ) { $r = &$ret[$n++]; $r = new Isis_Rec(); $r->db = $db; $r->head = $this->val[$p]; for ( $i=-(int)$t-1; $i-- && list($p,$t) = each($this->tag); ) { $r->tag[] = $t; $r->val[] = $this->val[$p]; } } return $ret; } /** same as v($tag,'&'.$fmt) */ function h ( $tag = null, $fmt = '' ) { return $this->v($tag,'&'.$fmt); } /** same as v($tag,'&MHL') */ function mhl ( $tag = null ) { return $this->v($tag,'&MHL'); } // //////////////////////////////////////////////////////////// // setters // /** append a new field (tag-value-pair) to the end of the record. @param int $tag tag to use in the field. fdt is applied @param string $val the new fields value. The string type is not enforced here. @return the new value */ function append ( $tag, $val ) { if ( !is_int($tag) ) $this->fdt($tag); // echo "0\tappending $tag ",gettype($val),"\n"; if ( is_string($val) || is_numeric($val) ) { $this->tag[] = $tag; $this->val[] = $val; } elseif ( is_array($val) ) { foreach ($val as $v) $this->append($tag, $v); } elseif ( is_object($val) ) $this->embed($val); return $val; } /** add an array to the record. @param mixed $argv an array either as a single parameter or as a variable number of arguments. The array is processed as follows: - if an item is an int, it is appended, with the following item as value. - if an item is an array, add is called recursively on this array - if an item is '-db' or '-mfn', the corresponding properties are set - if we have an fdt which maps the item to an int, the following item as value is added with the tag given by the fdt. - if an item is ISIS_REC_TEXT, the following item is parsed in text mode. - else the item is parsed in standard mode. */ function add ( $argv ) { $added = 0; $fdt = $this->db ? $this->db->fdt : null; if (1 < func_num_args() || !is_array($argv)) $argv = func_get_args(); // standardized to having an array for ( $i = reset($argv); $i || !is_bool($i); $i = next($argv) ) { if ( is_int($i) ) { if ( !is_null($this->append($i, next($argv))) ) $added++; } elseif ( is_array($i) ) $added += $this->add($i); elseif ( '-mfn' == $i ) $this->mfn = next($argv); elseif ( '-db' == $i ) { // they don't grok $fdt = ($this->db = next($argv))->fdt; $this->db = next($argv); $fdt = $this->db->fdt; } elseif ($fdt && is_int($fdt[$i])) { if ( !is_null($this->append($fdt[$i], next($argv))) ) $added++; } elseif ( ISIS_REC_TEXT == $i ) $added += $this->parse( next($argv), ISIS_REC_TEXT ); else $added += $this->parse( $i ); } } function addglobals () { if ( $this->db && $this->db->fdt ) foreach ($this->db->fdt as $name => $tag) $this->append($tag, $GLOBALS[$name]); } /** recompact after unsetting */ function pack () { $this->tag = array_values( $this->tag ); $this->val = array_values( $this->val ); } /** remove a field at given pos, poking a hole in the field list. */ function rm ( $pos, $pack = FALSE ) { unset( $this->tag[$pos] ); unset( $this->val[$pos] ); if ( $pack ) $this->pack(); } /** remove all fields or all with a given tag. */ function del ( $tag = null, $pack = FALSE ) { if ( is_null($tag) ) { $this->tag = array(); $this->val = array(); } else { if ( !is_int($tag) ) $this->fdt($tag); foreach ($this->tag as $p => $t) if ($tag == $t) $this->rm($p); if ( $pack ) $this->pack(); } } /** set fields with tag to values. set( 42, 'foo', 'bar', 'baz' ) will change the first three occurences of 42 to 'foo', 'bar' and 'baz', resp. - if there are less than three occurences, the remaining values are appended - if there are more than three occurences, the remaining occurences are deleted - if a value is the integer 0, processing stops (i.e. remaining occurences are left unchanged) - if a value is a positive integer n, processing skips n occurences (letting them unchanged) - if a value is an array, it's elements are used (it is flattened out non-recursively) @param mixed $tag tag by int or name @param mixed values... variable number of values */ function set ( $tag ) { if ( !is_int($tag) ) $this->fdt($tag); $argv = func_get_args(); reset($argv); each($argv); // eat $tag reset($this->tag); $ary = null; $more = 1; for (;;) { if ( $ary ) { if ( !(list($k,$arg) = each($ary)) ) { $ary = null; continue; } } else { if ( !(list($k,$arg) = each($argv)) ) break; if ( is_array($arg) ) { $ary = $arg; reset($ary); continue; } } // echo "setting '$arg'\n"; if (is_int($arg)) { if (!$arg) return; while ($more && 0 < $arg--) while (($more = (list($k,$v) = each($this->tag))) && $tag != $v) ; continue; } // now arg is a value to set/add if ($more) { while (($more = (list($k,$v) = each($this->tag))) && $tag != $v) ; if ($more) { $this->val[$k] = $arg; continue; } } $this->append($tag,$arg); } if ( $more ) while (list($k,$v) = each($this->tag)) if ($tag == $v) $this->rm($k); } /* transparently embed a record. */ function embed ( $that ) { $this->append( -($i = $that->len())-1, $that->head ); for ( $t = reset($that->tag), $v = reset($that->val); $i--; $t = next($that->tag), $v = next($that->val) ) { $this->tag[] = $t; $this->val[] = $v; } } // //////////////////////////////////////////////////////////// // conversion to/from other representations // /** serialize record to a string. After each field, including the last one, a newline is added. @param string $mode replacement value for newlines. suggested is one of the predefined constants. defaults to ISIS_REC_TEXT. @return the string representation of the record */ function toString ( $mode = ISIS_REC_TEXT ) { $s = ''; if ( strlen($this->head) ) { if ( '0'<=$this->head{0} && $this->head{0}<='9' ) $s .= "W\t"; $s .= $this->head."\n"; } // why in heaven is next a pre-increment ??? $t = reset($this->tag); $v = reset($this->val); for ( $i = count($this->tag); $i--; $t = next($this->tag), $v = next($this->val) ) $s .= $t . "\t" . str_replace("\n",$mode,$v) . "\n"; return $s; } /** parse text as record fields to add. For each non-empty line, initial digits are used as tag (empty == 0), an optional following tab is skipped, and the rest used as value, after replacing $repl, if given, with newlines. If the line starts with a tab and the record is not empty, a newline and the value are appended to the last field, else a new field is appended to the record. @param string $repl string to be converted back to newlines. use ISIS_REC_TEXT, if you know text is from toString(ISIS_REC_TEXT) @return number of fields added */ function parse ( $text, $repl = null ) { // need compact array in order to reliably know last index $conv = 0; $lines = explode("\n",$text); if ( $lines && strlen($lines[0]) ) { $line = $lines[0]; if ( $line{0}<'0' || '9'<$line{0} ) { $this->head = $line; array_shift($lines); } } foreach ($lines as $line) { if ('' == $line) // blank line or trailing newline continue; $dig = strspn($line,'0123456789-'); $t = $dig ? (int)substr($line,0,$dig) : 0; $v = substr( $line, $dig + ("\t" == $line{$dig} ? 1 : 0) ); if ( $repl ) $v = str_replace($repl,"\n",$v); $this->tag[] = $t; $this->val[] = $v; $conv++; } return $conv; } // parse /** return the "data fork" of this record by mapping a function to tags and values in parallel. @param function $func defaults to null, resulting in an array of fields, each an array [0] => $tag, [1] => $val. @return a new array as of array_map */ function map ($func = null) { return array_map($func,$this->tag,$this->val); } } // class Isis_Rec ?> php/Isis/Server.php0100644000076400001440000000460010145101677013363 0ustar gabiusershost = $host; $this->port = $port; $this->pers = $pers; $this->open(); } function open () { // pfsockopen will reuse an existing socket on same host:port $this->sock = $this->pers ? pfsockopen( $this->host, $this->port /*, $errno, $errstr*/ ) : fsockopen( $this->host, $this->port /*, $errno, $errstr*/ ); return $this->sock; } function request ( $req ) { if ( !$this->sock && !$this->open() ) return null; if ($this->dbg) error_log("SEND\n".$req->toString(ISIS_REC_TEXT)); fwrite($this->sock, $req->toString(ISIS_REC_TEXT) . "\n"); fflush($this->sock); $txt = ''; while ( ($line = fgets($this->sock)) && "\n" != $line ) { if ($this->dbg) error_log("RETR ".$line ); $txt .= $line; } $res = new Isis_Rec; $res->parse($txt, ISIS_REC_TEXT); if ($this->dbg) error_log("GOT ".$res->toString() ); return $res; } } // class Isis_Server ?> php/demo.php0100644000076400001440000001432710141707512012135 0ustar gabiusers Demo for package Isis 22, 'b' => 42 ) ); // create a db with field list ("fdt") $fdt = array( 'title' => 24, 'author' => 70, 'keywords' => 69 ); $db = new Isis_Db($fdt, 'test'); // a special value using lotsa subfields $subs = 'initial aParis bUnesco b c-1965'; // create record using various add parameters $r = new Isis_Rec( // set the db '-db', $db, // first some lines from CDS, some using field names, some plain int tags 'keywords', 'Paper on: ', 'author', 'Magalhaes, A.C.', 24, ' Controlled climate in the plant chamber', 76, 'Les Politiques de la communication en Yougoslavie zfre', 'author', 'Franco, C.M.', 26, $subs, // a field to test del 77, 'ave Caesar', // a field using tab as subfield separator 42, "foo\tbar\tbaz", // a field containing newline 99, "two\nlines", // a serialized record (as of toString) as parameter "70\tyet another author\n99two more\n99lines\na 0 field\n42\tthe\tanswer" ); // dump the stuff ?>

dump of record

have len()?> fields 
--
toString())?>--

examples of field selector

tag); // print_r corrupts data with 4.3.1 // print_r($r->map()); $r->del( 77 ); // ... morituri te salutant // various examples how to access fields foreach ($db->fdt as $name => $tag) while ($v = $r->mhl($tag)) echo "$name = $v
\n"; ?> v26^a = h(26,'a')?>
res(); // more convenient to get them all at once // the same as Isis_rec::fmt($r->v(26),'&abc'), foreach ($r->h(26,'abc') as $k => $v) echo "v26^$k = $v
\n"; ?> v69 = mhl('keywords')?>
v76 = mhl(76)?>
v42(1) = h(42,'(1)')?> formats returning arrays on '$subs'\n"; foreach (array( '&cab(..', '&*(..', '&(..', '&bca(1..', '&MHLcab(..' ) as $fmt) { echo "\n\n

Format "$fmt"

\n"; foreach (Isis_rec::fmt($subs,$fmt) as $k => $v) echo "v26^$k = $v
\n"; } ?>

embedding and TEXT mode

embed( $r ); $s = $q->toString(ISIS_REC_TEXT); echo "

record embedded

\n
---\n$s---\n
\n"; $q->del(); // restore from the string $q->parse( $s, ISIS_REC_TEXT ); // extract the original r $recs = $q->recs(); $r = $recs[0]; $r->db = $db; echo "

record restored from field 22

\n", "
---\n", $r->toString(), "---\n
\n"; ?>

set operator

set( 'title', 'new title', 'second new title' ); // change 99s, removing one $r->set( 99, 'now a oneliner' ); // change author[1] only, also test array parameter $r->set( 'author', array(1, 'Blanco', 0) ); echo "
---\n", $r->toString(), "---\n
\n"; ?>

server

srv->sock ) echo "could not contact server\n"; else { ?>

terms beginning with a

terms('a'); echo "got ",count($terms), "terms for 'a'
\n"; foreach ($terms as $t) { list($cnt, $term) = explode("\t", $t); echo "'$term'($cnt) "; } echo "
\n"; ?>

query reading records

query('plant water'); echo "got ",count($recs), "records for 'plant water'
\n"; foreach ($recs as $r) echo "
---\n", $r->toString(), "---\n
\n"; ?>

query reading mfns

query('plant + water + devel$', false); do { echo "got ",count($mfns), " mfns: ", join(',',$mfns),"
\n"; } while ($mfns = $db->query(null,false)); ?>

reading 42, 43

read( array(42,43) ); foreach ($recs as $r) echo "
---\n", $r->toString(), "---\n
\n"; ?>

reading 42

read(42); echo "
---\n", $r->toString(), "---\n
\n"; ?>

writing 42

append('author', 'one more author'); echo "
---\n", $r->toString(), "---\n
\n"; $mfns = $db->write($r); echo "wrote ",count($mfns), " mfns: ", join(',',$mfns),"
\n"; ?>

writing 42 as new record

head = ''; $mfns = $db->write($r); echo "wrote ",count($mfns), " mfns: ", join(',',$mfns),"
\n"; ?>

indexing author fields as 70 in split mode

head = 's'; $idx->set( 70, $r->get('author') ); echo "
---\n", $idx->toString(), "---\n
\n"; $res = $db->index($idx); echo "got $res
\n"; ?>

query one near author

query('one .. author', false); echo "got ",count($mfns), " mfns: ", join(',',$mfns),"
\n"; ?> php/killidx.php0100755000076400001440000000130510233444234012645 0ustar gabiusers#!/usr/bin/php srv->sock ) { echo "could not contact server\n"; return; } $mersenne7 = 0x7ffff; // 7th mersenne prime $mask = 0x7fffff; $quot = 0x77ffff; for ($i=1; $i<=100000; $i++) { // just to have some other number, from which we can compute i $j = $mask & ($mersenne7 * $i); // if ($i != ($mask & ($quot*$j))) { echo "$i $j broken\n"; return; } $idx = new Isis_Rec(); $idx->head = 'r'.$i."\tf"; $idx->append(1, sprintf("%x", $i)); $idx->append(2, sprintf("%x", $j)); $res = $db->index($idx); if (!($i & 0xff)) echo "$i\t$res\n"; } ?> php/killqry.php0100755000076400001440000000145710233540153012701 0ustar gabiusers#!/usr/bin/php srv->sock ) { echo "could not contact server\n"; return; } $mersenne7 = 0x7ffff; // 7th mersenne prime $mask = 0x7fffff; $quot = 0x77ffff; for ($i=1; $i<=10000; $i++) { if (!($i & 0xff)) echo "$i\n"; // just to have some other number, from which we can compute i $j = $mask & ($mersenne7 * $i); // $idx->append(1, sprintf("%x", $i)); // $idx->append(2, sprintf("%x", $j)); $recs = $db->query(sprintf("%x/1", $i), false); if (1 != count($recs) || $i != $recs[0]) echo "$i/1\t".join(",", $recs)."\n"; $recs = $db->query(sprintf("%x/2", $j), false); if (1 != count($recs) || $i != $recs[0]) echo "$i/2\t".join(",", $recs)."\n"; } ?> php/unicode.php0100644000076400001440000002157510145101725012640 0ustar gabiusersß< angle brackets, then: - use an UTF-8 capable xterm like LC_CTYPE=en_US.UTF-8 LESSCHARSET=UTF-8 \ xterm -u8 -fn -misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1 - use an UTF-8 capable editor like vim '+set encoding=utf-8' unicode.php moreover this file contains some german: stadt=city land=country fluss=river bitte=please suchen=search */ header("Content-type: text/html; charset=UTF-8"); require 'Isis.php'; // use require 'Isis/Rec.php' if you need only this ?> Unicode-Demo for package Isis 10, 'nach' => 11, 'mfn' => 12, 'stadt' => 21, 'land' => 22, 'fluss' => 23 ); // get request parameters a and b // (as well as any plain numeric and v%d style $db = new Isis_Db($fdt, 'unicode', new Isis_Server()); if ( !$db->srv->sock ) { echo "could not contact server\n"; exit; } // $db->srv->dbg = true; $param = Isis_Http::fromReq($db); $paramdump = $param->toString(); $bitte = $param->v('bitte'); $param->del('bitte'); $nach = $param->v('nach'); if ('\"' == substr($nach,0,2)) // grrr ... stupid magic_quotes_gpc $nach = stripslashes($nach); $param->del('nach'); $mfn = $param->v('mfn'); $param->del('mfn'); // if ('bearbeiten' == $bitte) { $param = $db->read($mfn); } ?> Isi
Stadt Land Fluß
nach alle suchen terms
write($param); echo "wrote ",count($mfns), " mfns: ", join(',',$mfns),"
\n"; $idx = new Isis_Rec(); $idx->head = 's'; $idx->set($fdt['stadt'], $param->v('stadt')); $idx->set($fdt['land'], $param->v('land')); $idx->set($fdt['fluss'], $param->v('fluss')); // echo "
---\n", $idx->toString(), "---\n
\n"; $res = $db->index($idx); echo "got $res
\n"; break; case 'suchen': if (!$nach) $nach = '""$'; $recs = $db->query($nach); ?>
this is the textbody
of the td node
\n"; list($mfn,) = split("@",$r->head); printf("\n", $mfn); printf("\n", $r->h('stadt')); printf("\n", $r->h('land')); printf("\n", $r->h('fluss')); echo "\n"; } echo "
Mfn Stadt Land Fluß
%s%s%s%s
\n"; break; case 'terms': $terms = $db->terms($nach); $nterms = count($terms); $percol = (int)(($nterms+4)/5); ?>
Indexeinträge
%s (%d)'."
\n", urlencode($term), $term, $cnt); if (!(++$n % $percol) && $nterms != $n) echo "		"; } echo "
\n"; break; } ?>
Code   Kategorie Name
aus Latin-1 Supplement 0080-00FF
00D6'Ö' LATIN CAPITAL LETTERO WITH DIAERESIS (Ouml)
00F1'ñ' LATIN SMALL LETTERN WITH TILDE (ntilde)
00F3'ó'O WITH ACUTE (oacute)
00F6'ö'O WITH DIAERESIS (ouml)
00FC'ü'U WITH DIAERESIS (uuml)
aus Latin Extended-A 0100-017F
010C'Č' LATIN CAPITAL LETTERC WITH CARON
0141'Ł'L WITH STROKE
0142'ł' LATIN SMALL LETTERL WITH STROKE
017A'ź'Z WITH ACUTE
aus IPA Extensions 0250-02AF
0283'ʃ' LATIN SMALL LETTERESH
028C'ʌ'TURNED V
02A8'ʨ'TC DIGRAPH WITH CURL
aus Greek and Coptic 0370-03FF
0391'Α' GREEK CAPITAL LETTERALPHA (Alpha)
0395'Ε'EPSILON (Epsilon)
03AC'ά' GREEK SMALL LETTERALPHA WITH TONOS
03AE'ή'ETA WITH TONOS
03B1'α'ALPHA (alpha)
03B4'δ'DELTA (delta)
03B8'θ'THETA (theta)
03BB'λ'LAMDA (lambda)
03BD'ν'NU (nu)
aus Cyrillic 0400-04FF
041C'М' CYRILLIC CAPITAL LETTEREM
0420'Р'ER
0430'а' CYRILLIC SMALL LETTERA
0432'в'VE
0438'и'I
043A'к'KA
043E'о'O
0441'с'ES
044F'я'YA
html ', $aw); printf('pdf', $beg); printf("
\n"); for (; $beg < $end; ) { $line = sprintf("%04X", $beg); for ($i=16; $i--; $beg++) $line .= sprintf(" %c%c", 192+($beg>>6), 128+(63&$beg)); printf("%s
\n", $line); } } // c.f. http://openisis.org/Doc/UniStat // http://www.alanwood.net/unicode/#links printrange('Latin-1 Supplement', 0x0080, 0x00FF, 'latin_1_supplement'); printrange('Latin Extended-A', 0x0100, 0x017F, 'latin_extended_a'); printrange('Latin Extended-B', 0x0180, 0x024F, 'latin_extended_b'); printrange('IPA Extensions', 0x0250, 0x02AF, 'ipa_extensions'); printrange('Greek and Coptic', 0x0370, 0x03FF, 'greek'); printrange('Cyrillic', 0x0400, 0x04FF, 'cyrillic');
Parameters: 
--
--
*/ ?> test/cds.m0d0100644000076400001440000000066310144726513012046 0ustar gabiusers# $Id: cds.m0d,v 1.3 2004/11/11 18:19:55 kripke Exp $ 4 C spanish 4 N . ; : , - ( ) / $ 4 W 0 1 2 3 4 5 6 7 8 9 4 W a b c ch d e f g h i j k l ll m n o p q r s t u v w x y z 4 A A B C CH D E F G H I J K L LL M N O P Q R S T U V W X Y Z 4 M a 4 M ae 4 M c 4 M e 4 M i 4 M th 4 M o 4 M oe 4 M u 4 M ue 4 M y 4 M ss test/cds.mqt0100644000076400001440000014743310006472663012177 0ustar gabiusersABBAS, B.M. 59 70 1 1 ABEYWICKRAMA, B.A. 61 70 1 1 ABOUT 103 24 1 1 ABSENCE 43 24 1 9 ABSORPTION 6 24 1 10 ACCOMPANYING 37 24 1 8 ACCOUNT 88 24 1 5 ACCOUNTING 137 69 1 1 ACHIEVEMENTS 47 24 1 2 ACID 41 24 1 6 ACTION 102 24 1 5 ACTIVITIES 50 24 1 6 ACTIVITIES 131 24 1 7 ACTUAL 24 24 1 5 ADDRESSES 138 24 1 1 ADMINISTRATIVE REFORM 92 69 1 8 ADMISSION 123 69 1 8 ADMISSION 124 69 1 9 ADMISSION CRITERIA 88 69 1 7 ADULT 84 24 1 1 ADULT EDUCATION 84 69 1 1 ADULT EDUCATION 117 69 1 1 ADULT EDUCATION 146 69 1 1 ADULT EDUCATION 149 69 1 11 AERIAL 34 24 1 4 AERIAL PHOTOGRAPHY 31 69 1 6 AERIAL PHOTOGRAPHY 34 69 1 4 AESTHETICS 145 69 1 3 AFFECTIVE 82 24 1 5 AFGHANISTAN 134 69 1 5 AFRICA 97 69 1 9 AFRICA 115 24 1 3 AFRICA 115 69 1 3 AFRICA 119 69 1 7 AFRICA 125 69 1 4 AFRICAN LANGUAGES 79 69 1 10 AGE 140 24 1 7 AGE GROUPS 120 69 1 9 AGENT 55 24 1 5 AGENTS 85 24 1 4 AGRICULTURAL DEVELOPMENT 50 69 1 5 AGRICULTURAL DEVELOPMENT 71 69 1 3 AGRICULTURAL DEVELOPMENT 130 69 1 1 AGRICULTURAL TRAINING 84 69 1 10 AGRICULTURE 17 69 1 3 AGRICULTURE 65 69 1 3 AGRICULTURE 69 69 1 5 AGRICULTURE 70 69 1 6 AGRICULTURE 71 24 1 1 AGRICULTURE 73 69 1 2 AGRICULTURE 74 69 1 7 AGRONOMY 31 69 1 7 AHMAD, MAJEED 60 70 1 1 AHMAD, NAZIR 67 70 1 1 AHMAD, NAZIR 73 70 1 1 ALLUVIAUX 29 24 1 13 ALLUVIONS 30 24 1 7 ALTERNATIVE 141 24 1 8 AMAZON 37 24 1 18 AMAZON 39 24 1 7 AMAZON 62 24 1 5 AMAZON 75 24 1 8 AMERICA 105 24 1 8 ANALYSIS 46 24 1 4 ANALYSIS 110 24 1 7 ANALYTICAL 122 24 1 1 ANAYA, RICARDO 128 70 1 1 ANCIENT CIVILIZATIONS 89 69 1 1 ANIMAL 70 24 1 3 ANNA 129 24 1 9 ANNOTATED 115 24 1 9 ANNUAL 109 24 1 1 ANTI 5 24 1 1 ANTIQUITIES 89 69 1 3 APERCU 30 24 1 1 APPARATUS 4 24 1 4 APPROPRIATE TECHNOLOGY 141 69 1 6 APRIL 138 24 1 17 APTER, DAVID ERNEST 111 70 1 1 AQUIFERS 57 69 1 4 ARAB COUNTRIES 119 69 1 3 ARAB COUNTRIES 120 69 1 3 ARAB COUNTRIES 123 69 1 4 ARAB COUNTRIES 124 69 1 4 ARABES 120 24 1 9 ARABES 123 24 1 12 ARABES 124 24 1 10 ARCHITECTURE 94 69 1 6 ARCHITECTURE 103 69 1 10 AREA 37 24 1 19 AREA 37 24 1 24 AREA 46 24 1 3 AREA 74 24 1 3 AREAS 24 24 1 9 AREAS 58 24 1 5 AREAS 80 24 1 8 ARID 22 24 1 11 ARMAMENTS 140 24 1 1 ARMS CONTROL 140 69 1 5 ARMS SALES 140 69 1 6 ART BOOKS 89 69 1 6 ART HISTORY 89 69 1 4 ARTISTIC CREATION 79 69 1 11 ARTS 145 69 1 4 ASIA 97 69 1 7 ASIA 134 69 1 4 ASPECTS 123 24 1 1 ASPECTS 129 24 1 6 ASPIRATION 86 69 1 20 ASSIMILATION 2 24 1 12 ASSURANCES 101 24 1 4 ASSURANCES 151 24 1 4 AT 119 24 1 18 AT 138 24 1 3 ATANGANA, ENGELBERT 79 70 1 2 ATTITUDE 95 69 1 22 AUDIT 137 24 1 2 AUSTRALIA 146 69 1 2 AUSTRALIAN 146 24 1 1 AUTOMATIC 3 24 1 9 AVAILABLE 133 24 1 3 AVAILABLE 134 24 1 3 AWAY 90 24 1 14 AYALA, FRANCISCO JOSE 99 70 1 1 BAEZ, ALBERT V. 87 70 1 1 BAHOKEN, J.C. 79 70 1 1 BALANCE 14 24 1 6 BALANCE 24 24 1 16 BANGLADESH 35 69 1 4 BANGLADESH 38 69 1 7 BANGLADESH 40 69 1 7 BANGLADESH 42 69 1 7 BANGLADESH 58 69 1 7 BANGLADESH 59 69 1 6 BANGLADESH 60 69 1 7 BANGLADESH 64 69 1 7 BANGLADESH 66 69 1 7 BANGLADESH 67 69 1 6 BANGLADESH 69 69 1 6 BANGLADESH 70 69 1 9 BANGLADESH 72 69 1 6 BANGLADESH 73 69 1 6 BANGLADESH 78 69 1 12 BARRS, H.D. 10 70 1 2 BARRS, H.D. 13 70 1 1 BARS 36 24 1 7 BASED 149 24 1 5 BASEL 98 24 1 12 BASIC 92 24 1 9 BASIC 147 24 1 4 BASIC EDUCATION 92 69 1 2 BASIS 51 24 1 3 BAUMGARTNER, ALBERT 25 70 1 1 BE 119 24 1 16 BEHAVIOUR 5 24 1 18 BELOW 18 24 1 10 BENGAL 68 24 1 12 BETWEEN 144 24 1 3 BIBLIOGRAPHIE 98 24 1 1 BIBLIOGRAPHIES 115 69 1 6 BIBLIOGRAPHY 115 24 1 10 BIOLOGICAL EVOLUTION 99 69 1 5 BIOLOGICAL RESEARCH 99 69 1 3 BIOLOGY 28 69 1 8 BIOLOGY 52 24 1 2 BIOLOGY 52 69 1 3 BIOLOGY 53 24 1 1 BIOLOGY 53 69 1 1 BIOLOGY 68 69 1 2 BIOLOGY 70 69 1 3 BIOLOGY 99 24 1 6 BIOLOGY 99 69 1 1 BIOMASS 26 69 1 3 BIOMASS 141 69 1 12 BITSIOS, DIMITRI S. 107 70 1 1 BJELICA, MIHALO 91 70 1 2 BLACKS 95 69 1 15 BLACKS 115 69 1 2 BLOOM, JOSEPH 90 70 1 2 BOARDING SCHOOLS 90 69 1 2 BOESCH, H. 50 70 1 1 BOLIVIA 128 69 1 3 BONDURANT, JOAN V. 142 70 1 1 BOOK 84 24 1 6 BORDIA, ANIL 84 70 1 1 BORICHANSKY, L.S. 43 70 1 1 BORING 68 24 1 2 BOROBUDUR 94 24 1 2 BOSIAN, G. 2 70 1 1 BOSIAN, G. 3 70 1 1 BOURDEAU, PHILIPPE F. 6 70 1 1 BOURDEAU, PHILIPPE F. 27 70 1 2 BOWRA, SIR CECIL MAURICE 89 70 1 1 BRACKISH 52 24 1 5 BRACKISH WATER 49 69 1 3 BRACKISH WATER 52 69 1 4 BRACKISH WATER 57 69 1 7 BRACKISH WATER 66 69 1 5 BRAIN DRAIN 125 69 1 11 BRASIL 108 24 1 4 BRAZIL 37 69 1 6 BRAZIL 39 69 1 8 BRAZIL 62 69 1 7 BRAZIL 75 69 1 6 BRAZIL 105 69 1 8 BRAZIL 108 69 1 2 BRINKMAN, GEORGE LORRIS 100 70 1 2 BRINKMAN, GEORGE LORRIS 150 70 1 2 BRIX, H. 11 70 1 2 BROADCAST 93 24 1 1 BROADCASTING 93 69 1 9 BROADCASTING 96 24 1 4 BROTHERS 129 24 1 12 BUDDHISM 94 69 1 5 BUDEL, J. 51 70 1 1 BUDGET 25 24 1 7 BUILDING 35 24 1 14 BURMA 58 69 1 8 BURMA 74 24 1 9 BURMA 74 69 1 10 BURMA 78 69 1 9 BUSINESS MANAGEMENT 110 69 1 2 C 113 24 1 4 CALCULATION 24 24 1 1 CAMEROON 79 24 1 8 CAMEROON UR 79 69 1 2 CANADA 117 69 1 4 CARACTERISTIQUES 29 24 1 3 CARBON 6 24 1 4 CARBON 25 24 1 5 CARBON DIOXIDE 7 69 1 2 CARBON DIOXIDE 25 69 1 3 CARBON DIOXIDE 26 69 1 4 CARBON DIOXIDE 81 69 1 2 CARBONIQUE 81 24 1 10 CARIBBEAN AREA 115 69 1 4 CARLIER, G. 81 70 1 2 CARTE 104 24 1 13 CARTOGRAPHY 31 69 1 5 CARTOGRAPHY 48 69 1 6 CARTOGRAPHY 143 69 1 7 CASE STUDIES 86 69 1 1 CASE STUDIES 104 69 1 8 CASE STUDIES 149 69 1 7 CASE, JOHN H. 86 70 1 2 CASPERS, H. 53 70 1 1 CAT 41 24 1 9 CATALOGUES 115 69 1 7 CATSKY, J. 7 70 1 2 CATSKY, J. 12 70 1 1 CENTRE 106 24 1 10 CEREALS 49 69 1 9 CEREALS 71 69 1 1 CEREMONY 138 24 1 6 CEYLON 61 24 1 5 CHAMBER 2 24 1 7 CHAMBER 3 24 1 7 CHANCELLOR 88 24 1 10 CHANDI 94 24 1 1 CHANGE 85 24 1 3 CHANGE 111 24 1 5 CHANGES 56 24 1 2 CHANGING 143 24 1 4 CHANGING SOCIETY 143 69 1 2 CHAO 37 24 1 21 CHAO 78 24 1 16 CHAPMAN, VALENTINE J. 63 70 1 1 CHEMICAL ANALYSIS 7 69 1 3 CHEMICAL PROPERTIES 29 69 1 6 CHEMICAL PROPERTIES 39 69 1 4 CHEMICAL PROPERTIES 64 69 1 5 CHEMICAL/BIOLOGICAL WARFARE 140 69 1 4 CHI 78 24 1 15 CHILDREN 90 24 1 13 CHILDREN'S BOOKS 115 69 1 8 CHIMIQUE 33 24 1 3 CHO 78 24 1 13 CHOWDHURY, M.I. 35 70 1 1 CHURCH 108 69 1 8 CHURCHILL, STACY 92 70 1 1 CIVILIZATION 51 24 1 7 CIVILIZATION 51 69 1 3 CLASSICAL 89 24 1 1 CLASSIFICATION 28 69 1 10 CLASSIFICATION 45 69 1 2 CLASSIFICATION 76 24 1 2 CLASSIFICATION 76 69 1 2 CLAYS 41 24 1 10 CLIMATE 2 24 1 3 CLIMATE 15 69 1 4 CLIMATE 18 69 1 4 CLIMATE 46 69 1 2 CLIMATE 64 69 1 3 CLIMATE 76 69 1 4 CLIMATIC 37 24 1 14 CLIMATIC 76 24 1 9 CLIMATIC DATA 42 69 1 5 CLUBS 95 69 1 19 CO 7 24 1 4 COASTAL 77 24 1 5 COASTAL ZONES 57 69 1 5 COASTAL ZONES 63 69 1 6 COASTAL ZONES 68 69 1 6 COASTAL ZONES 77 69 1 6 COLOMBIA 105 69 1 7 COLORIMETRIC 7 24 1 1 COMMUNICATION 91 24 1 1 COMMUNICATION 135 24 1 1 COMMUNICATION HISTORY 91 69 1 9 COMMUNICATION IMPACT 108 69 1 6 COMMUNICATION LEGISLATION 91 69 1 15 COMMUNICATION PERSONNEL TRAINI 93 69 1 1 COMMUNICATION PLANNING 96 69 1 1 COMMUNICATION POLICY 91 69 1 1 COMMUNICATION PROCESS 82 69 1 6 COMMUNICATION PROCESS 135 69 1 2 COMMUNICATION RESEARCH 135 69 1 4 COMMUNICATION STATISTICS 91 69 1 20 COMMUNICATION STATISTICS 113 69 1 13 COMMUNITIES 95 69 1 6 COMMUNITY 95 24 1 9 COMMUNITY DEVELOPMENT 100 69 1 8 COMMUNITY DEVELOPMENT 150 69 1 8 COMMUNITY EDUCATION 84 69 1 6 COMMUNITY EDUCATION 127 69 1 11 COMPARATIVE 78 24 1 1 COMPARATIVE ANALYSIS 82 69 1 3 COMPARISON 76 24 1 4 COMPULSORY EDUCATION 104 69 1 10 COMPULSORY EDUCATION 120 69 1 12 COMPULSORY EDUCATION 124 69 1 5 COMPUTER APPLICATIONS 83 69 1 3 COMPUTER APPLICATIONS 137 69 1 2 COMPUTER HARDWARE 137 69 1 5 COMPUTER SOFTWARE 137 69 1 6 COMUNICACAO 108 24 1 1 CONCENTRATION 33 24 1 2 CONCLUSIONS 132 24 1 1 CONDITIONS 3 24 1 3 CONDITIONS 3 24 1 23 CONDITIONS 6 24 1 13 CONDITIONS 37 24 1 15 CONDITIONS 77 24 1 14 CONDITIONS 81 24 1 18 CONFERENCE 124 24 1 19 CONFERENCES 119 24 1 7 CONFERENCES 119 69 1 1 CONFLICT 142 24 1 8 CONFLICT RESOLUTION 142 69 1 2 CONFORM 3 24 1 19 CONQUEST 142 24 1 1 CONSULTANCY 133 24 1 1 CONTEMPORARY SOCIETY 103 69 1 2 CONTEMPORARY SOCIETY 121 69 1 3 CONTRIBUTES 52 24 1 11 CONTROL 3 24 1 1 CONTROL 59 24 1 1 CONTROLLED 2 24 1 2 CONTROLLED 6 24 1 12 CORPORATION 111 24 1 2 COUGHLAN, MARGARET N. 115 70 1 1 COUNTRIES 113 24 1 8 COUNTRIES 133 24 1 7 COUNTRIES 134 24 1 7 COUNTRIES 141 24 1 12 COUNTRIES 149 24 1 12 COUNTRY REPORTS 105 69 1 21 COURSES 134 24 1 2 COVER 21 24 1 9 COVER 25 24 1 10 COVER 27 24 1 9 COVERS 22 24 1 8 CRAIN, N. 46 70 1 1 CRITICISM 129 24 1 24 CROISSANCE 120 24 1 2 CROP 17 24 1 6 CROPPED 24 24 1 8 CROSS 82 24 1 1 CUBA 113 69 1 4 CULTIVATION 49 69 1 10 CULTIVATION 50 69 1 4 CULTIVATION 71 24 1 7 CULTIVATION 71 69 1 2 CULTURAL 79 24 1 1 CULTURAL 82 24 1 2 CULTURAL ACTIVITIES 79 69 1 12 CULTURAL DEVELOPMENT 51 69 1 2 CULTURAL DEVELOPMENT 108 69 1 4 CULTURAL FACILITIES 79 69 1 13 CULTURAL FACTORS 82 69 1 7 CULTURAL HISTORY 89 69 1 2 CULTURAL IDENTITY 79 69 1 5 CULTURAL INSTITUTIONS 79 69 1 3 CULTURAL PLANNING AND ADMINIST 79 69 1 4 CULTURAL POLICY 79 69 1 1 CULTURAL POLICY 119 69 1 12 CULTURAL PROGRAMMES 102 69 1 1 CULTURE 51 24 1 5 CULTURE 65 24 1 8 CURRENT 106 24 1 4 CURRENTS 77 24 1 6 CURRICULUM DEVELOPMENT 93 69 1 7 CURRICULUM DEVELOPMENT 114 69 1 7 CURRICULUM DEVELOPMENT 121 69 1 1 CURRICULUM PLANNING 92 69 1 6 CYPRUS 107 24 1 1 CYPRUS 107 69 1 3 D 81 24 1 23 DACCA 28 24 1 15 DANCE, F. ROBERT 93 70 1 1 DANS 29 24 1 14 DANS 31 24 1 6 DANS 32 24 1 4 DANS 33 24 1 9 DANS 120 24 1 6 DANS 123 24 1 9 DANS 124 24 1 7 DATA COLLECTION 105 69 1 15 DATA PROCESSING 105 69 1 16 DAVE 122 24 1 14 DAVE, RAVINDRAKUMAR H. 122 70 1 2 DE 29 24 1 21 DE 31 24 1 3 DE 31 24 1 9 DE 32 24 1 2 DE 81 24 1 5 DE 81 24 1 8 DE 81 24 1 20 DE 98 24 1 3 DE 104 24 1 5 DE 104 24 1 11 DE 123 24 1 3 DE 124 24 1 3 DE 124 24 1 14 DE 124 24 1 17 DE 124 24 1 20 DECENTRALIZATION 92 69 1 7 DECISION 83 24 1 5 DECISION MAKING 83 69 1 1 DEFICIT 12 24 1 8 DEFINIES 81 24 1 19 DEFINITIONS 87 69 1 13 DEFINITIONS 104 69 1 6 DEGRE 123 24 1 8 DEGREES 88 69 1 2 DEGUCHI, K. 49 70 1 1 DELIVERED 138 24 1 2 DELTA 29 24 1 16 DELTA 34 24 1 7 DELTA 35 24 1 13 DELTA 52 24 1 14 DELTA 56 24 1 8 DELTA 75 24 1 5 DELTAIC 42 24 1 6 DELTAIC 58 24 1 4 DELTAIC 74 24 1 2 DELTAIC 80 24 1 7 DELTAS 28 24 1 8 DELTAS 28 69 1 1 DELTAS 29 69 1 3 DELTAS 30 69 1 3 DELTAS 31 24 1 15 DELTAS 31 69 1 2 DELTAS 32 24 1 6 DELTAS 32 69 1 2 DELTAS 33 24 1 11 DELTAS 33 69 1 4 DELTAS 34 69 1 2 DELTAS 35 69 1 2 DELTAS 36 69 1 3 DELTAS 38 69 1 3 DELTAS 41 24 1 12 DELTAS 42 69 1 2 DELTAS 43 69 1 4 DELTAS 44 24 1 5 DELTAS 44 69 1 4 DELTAS 45 24 1 7 DELTAS 45 69 1 1 DELTAS 47 69 1 2 DELTAS 48 24 1 6 DELTAS 48 69 1 2 DELTAS 50 69 1 7 DELTAS 51 24 1 1 DELTAS 51 69 1 1 DELTAS 52 69 1 1 DELTAS 54 24 1 14 DELTAS 54 69 1 3 DELTAS 55 24 1 8 DELTAS 55 69 1 3 DELTAS 56 69 1 3 DELTAS 57 24 1 4 DELTAS 57 69 1 2 DELTAS 58 69 1 3 DELTAS 59 69 1 5 DELTAS 64 69 1 4 DELTAS 67 69 1 4 DELTAS 71 24 1 12 DELTAS 71 69 1 6 DELTAS 74 69 1 2 DELTAS 75 69 1 1 DELTAS 76 24 1 6 DELTAS 76 69 1 1 DELTAS 77 24 1 17 DELTAS 77 69 1 3 DELTAS 78 69 1 2 DELTAS 80 69 1 3 DEMAND 117 24 1 1 DEMOCRATIC KAMPUCHEA 30 69 1 7 DEMOCRATIZATION OF EDUCATION 128 69 1 6 DES 29 24 1 7 DES 29 24 1 11 DES 31 24 1 14 DES 33 24 1 6 DES 98 24 1 6 DES 101 24 1 3 DES 120 24 1 3 DES 151 24 1 3 DESARROLLO 128 24 1 1 DESCRIPTION 122 24 1 2 DESIGN 85 24 1 9 DETERMINATION 7 24 1 2 DETERMINATION 14 24 1 1 DETERMINATION 15 24 1 7 DETERMINATION 21 24 1 2 DETERMINING 12 24 1 5 DETERMINING 20 24 1 4 DEVELOPED COUNTRIES 147 69 1 13 DEVELOPING 133 24 1 5 DEVELOPING 134 24 1 5 DEVELOPING 141 24 1 11 DEVELOPING 149 24 1 11 DEVELOPING COUNTRIES 87 69 1 3 DEVELOPING COUNTRIES 111 69 1 4 DEVELOPING COUNTRIES 128 69 1 10 DEVELOPING COUNTRIES 130 69 1 8 DEVELOPING COUNTRIES 132 69 1 4 DEVELOPING COUNTRIES 141 69 1 7 DEVELOPING COUNTRIES 147 69 1 12 DEVELOPING COUNTRIES 149 69 1 3 DEVELOPMENT 100 24 1 2 DEVELOPMENT 100 24 1 10 DEVELOPMENT 114 24 1 2 DEVELOPMENT 130 24 1 7 DEVELOPMENT 131 24 1 17 DEVELOPMENT 141 24 1 4 DEVELOPMENT 149 24 1 20 DEVELOPMENT 150 24 1 2 DEVELOPMENT 150 24 1 10 DEVELOPMENT PLANNING 118 69 1 8 DEVELOPMENT PLANNING 127 69 1 14 DEVELOPMENT STRATEGIES 86 69 1 23 DEVELOPMENT STRATEGIES 147 69 1 4 DEVELOPPEMENT 124 24 1 1 DEW 18 24 1 4 DIFFERENT 40 24 1 4 DIFFUSION 110 24 1 13 DIOXIDE 6 24 1 5 DIOXIDE 25 24 1 6 DIPLOMAS 123 69 1 9 DIRECTOR 138 24 1 11 DIRECTORIES 133 69 1 6 DIRECTORIES 134 69 1 8 DIRECTORIES 141 69 1 16 DISARMAMENT 140 24 1 3 DISARMAMENT 140 69 1 2 DISC 12 24 1 2 DISCHARGE OF RIVERS 36 69 1 6 DISCHARGE OF RIVERS 59 69 1 2 DISCIPLINE 114 24 1 5 DISCUSSED 119 24 1 17 DISSEMINATION OF INFORMATION 91 69 1 5 DOBZHANSKY, THEODOSIUS GRIGORI 99 70 1 2 DOCUMENTATION 106 24 1 9 DOCUMENTATION CENTRES 106 69 1 1 DOCUMENTOS 118 24 1 4 DOTY, M.S. 52 70 1 1 DOW, PHILIP S. 86 70 1 3 DRAINAGE 49 69 1 7 DRAINAGE 59 69 1 1 DRAPER, JAMES A. 84 70 1 3 DROUGHT 16 24 1 4 DROUGHT 16 69 1 1 DROUGHT 17 24 1 3 DROUGHT 17 69 1 1 DRY 26 24 1 6 DRY 27 24 1 3 DU 29 24 1 17 DU 123 24 1 6 DU 129 24 1 17 DUE 56 24 1 6 DUKE, CHRISTOPHER 146 70 1 1 E 113 24 1 6 EAST 35 24 1 11 EAST 38 24 1 3 EAST 40 24 1 8 EAST 42 24 1 10 EAST 59 24 1 5 EAST 60 24 1 4 EAST 70 24 1 8 EAST 72 24 1 3 EAST 73 24 1 7 EASTERN EUROPE 113 69 1 3 EASTERN EUROPE 147 69 1 14 EAUX 33 24 1 7 ECOLOGY 68 69 1 3 ECOLOGY 70 69 1 4 ECONOMIC 100 24 1 9 ECONOMIC 150 24 1 9 ECONOMIC AND SOCIAL DEVELOPMEN 47 69 1 4 ECONOMIC AND SOCIAL DEVELOPMEN 111 69 1 7 ECONOMIC AND SOCIAL DEVELOPMEN 130 69 1 5 ECONOMIC AND SOCIAL DEVELOPMEN 132 69 1 1 ECONOMIC BEHAVIOUR 110 69 1 5 ECONOMIC CONCENTRATION 111 69 1 2 ECONOMIC CONDITIONS 95 69 1 14 ECONOMIC GROWTH 147 69 1 3 ECONOMIC POLICY 100 69 1 11 ECONOMIC POLICY 110 69 1 3 ECONOMIC POLICY 150 69 1 11 ECONOMICS OF EDUCATION 100 69 1 6 ECONOMICS OF EDUCATION 150 69 1 6 ECONOMY 143 69 1 6 ECOSYSTEMS 3 69 1 6 ECOSYSTEMS 27 69 1 3 EDUCATION 84 24 1 2 EDUCATION 85 24 1 14 EDUCATION 86 24 1 2 EDUCATION 87 24 1 4 EDUCATION 92 24 1 10 EDUCATION 122 24 1 9 EDUCATION 139 24 1 10 EDUCATION 146 24 1 5 EDUCATION AND DEVELOPMENT 79 69 1 8 EDUCATION AND DEVELOPMENT 119 69 1 2 EDUCATION AND DEVELOPMENT 128 69 1 1 EDUCATION AND EMPLOYMENT 86 69 1 7 EDUCATION AND EMPLOYMENT 95 69 1 13 EDUCATIONAL 114 24 1 1 EDUCATIONAL 136 24 1 2 EDUCATIONAL ADMINISTRATION 92 69 1 9 EDUCATIONAL ADMINISTRATION 136 69 1 3 EDUCATIONAL ADMINISTRATIVE STR 88 69 1 9 EDUCATIONAL ADMINISTRATIVE STR 92 69 1 5 EDUCATIONAL ADMINISTRATORS 127 69 1 10 EDUCATIONAL ADVISERS 85 69 1 1 EDUCATIONAL ASSISTANCE 127 69 1 2 EDUCATIONAL ASSISTANCE 149 69 1 1 EDUCATIONAL COSTS 87 69 1 23 EDUCATIONAL COSTS 104 69 1 19 EDUCATIONAL DEMAND 117 69 1 3 EDUCATIONAL DEVELOPMENT 114 69 1 1 EDUCATIONAL DEVELOPMENT 119 69 1 13 EDUCATIONAL DEVELOPMENT 120 69 1 2 EDUCATIONAL DEVELOPMENT 123 69 1 2 EDUCATIONAL DEVELOPMENT 124 69 1 2 EDUCATIONAL ENVIRONMENT 90 69 1 6 EDUCATIONAL ENVIRONMENT 93 69 1 14 EDUCATIONAL ENVIRONMENT 104 69 1 14 EDUCATIONAL EQUIPMENT 87 69 1 18 EDUCATIONAL EXPENDITURE 120 69 1 16 EDUCATIONAL EXPERIMENTS 149 69 1 9 EDUCATIONAL GOALS 87 69 1 8 EDUCATIONAL GOALS 104 69 1 9 EDUCATIONAL GOALS 128 69 1 5 EDUCATIONAL HISTORY 98 69 1 2 EDUCATIONAL INNOVATIONS 85 69 1 2 EDUCATIONAL INNOVATIONS 87 69 1 1 EDUCATIONAL INNOVATIONS 92 69 1 11 EDUCATIONAL INNOVATIONS 127 69 1 6 EDUCATIONAL INNOVATIONS 149 69 1 4 EDUCATIONAL INSTITUTIONS 86 69 1 17 EDUCATIONAL INSTITUTIONS 87 69 1 14 EDUCATIONAL LEVELS 120 69 1 11 EDUCATIONAL MANAGEMENT 114 69 1 11 EDUCATIONAL NEEDS 104 69 1 13 EDUCATIONAL NEEDS 149 69 1 6 EDUCATIONAL OUTPUT 120 69 1 8 EDUCATIONAL PERSONNEL TRAINING 85 69 1 6 EDUCATIONAL PLANNING 86 69 1 22 EDUCATIONAL PLANNING 92 69 1 10 EDUCATIONAL PLANNING 104 69 1 2 EDUCATIONAL PLANNING 114 69 1 3 EDUCATIONAL PLANNING 125 69 1 1 EDUCATIONAL PLANNING 128 69 1 4 EDUCATIONAL POLICY 84 69 1 2 EDUCATIONAL PROGRAMMES 102 69 1 2 EDUCATIONAL PROGRAMMES 114 69 1 6 EDUCATIONAL PROGRAMMES 136 69 1 2 EDUCATIONAL PSYCHOLOGY 90 69 1 7 EDUCATIONAL PSYCHOLOGY 148 69 1 1 EDUCATIONAL RADIO 88 69 1 4 EDUCATIONAL REFORM 87 69 1 9 EDUCATIONAL REFORM 92 69 1 1 EDUCATIONAL REFORM 104 69 1 12 EDUCATIONAL REFORM 128 69 1 9 EDUCATIONAL RESEARCH 97 69 1 3 EDUCATIONAL RESEARCH 114 69 1 4 EDUCATIONAL RESEARCH 121 69 1 6 EDUCATIONAL STATISTICS 84 69 1 13 EDUCATIONAL STATISTICS 86 69 1 10 EDUCATIONAL STATISTICS 88 69 1 12 EDUCATIONAL STATISTICS 90 69 1 9 EDUCATIONAL STATISTICS 104 69 1 17 EDUCATIONAL STATISTICS 114 69 1 12 EDUCATIONAL STATISTICS 117 69 1 7 EDUCATIONAL STATISTICS 120 69 1 4 EDUCATIONAL STATISTICS 123 69 1 5 EDUCATIONAL STATISTICS 124 69 1 6 EDUCATIONAL STRATEGIES 87 69 1 22 EDUCATIONAL STRATEGIES 139 69 1 3 EDUCATIONAL SUPERVISION 127 69 1 9 EDUCATIONAL SYSTEMS 120 69 1 14 EDUCATIONAL TECHNOLOGY 87 69 1 15 EDUCATIONAL TECHNOLOGY 127 69 1 8 EDUCATIONAL TECHNOLOGY 149 69 1 5 EDUCATIONAL TELEVISION 88 69 1 5 EDUCATIONAL TELEVISION 96 69 1 4 EDUCATIONAL TRENDS 87 69 1 10 EDUCATIONAL TRENDS 120 69 1 13 EDUCATIONAL TRENDS 146 69 1 4 EDUCATIVES 104 24 1 7 EFFECT 77 24 1 2 EFFECTIFS 120 24 1 4 EFFECTS 5 24 1 12 EFFECTS 90 24 1 6 EGOROV, V.V. 45 70 1 1 ELECTRIC 4 24 1 2 ELECTRICAL ENGINEERING 141 69 1 10 ELEMENT 37 24 1 9 ELKANA, YEHUDA 144 70 1 1 EMISSION 81 24 1 7 EMPLOYERS 86 69 1 21 EMPLOYMENT 86 69 1 5 EMPLOYMENT 100 69 1 5 EMPLOYMENT 125 69 1 10 EMPLOYMENT 147 24 1 1 EMPLOYMENT 147 69 1 2 EMPLOYMENT 150 69 1 5 EN 31 24 1 12 EN 81 24 1 17 EN 104 24 1 3 EN 120 24 1 14 ENERGY 109 24 1 4 ENERGY 109 69 1 1 ENERGY 141 24 1 1 ENERGY BALANCE 25 69 1 1 ENERGY CONVERSION 141 69 1 8 ENERGY RESOURCES 109 69 1 3 ENERGY RESOURCES 141 69 1 3 ENROLMENT 104 69 1 18 ENROLMENT 123 69 1 6 ENROLMENT 124 69 1 7 ENROLMENT PROJECTIONS 120 69 1 6 ENROLMENT PROJECTIONS 124 69 1 10 ENROLMENT RATIOS 120 69 1 5 ENSEIGNEMENT 124 24 1 5 ENSEIGNEMENT 124 24 1 24 ENVIRONMENTAL 139 24 1 3 ENVIRONMENTAL 139 24 1 9 ENVIRONMENTAL CHANGES 41 69 1 5 ENVIRONMENTAL CONTROL 109 69 1 8 ENVIRONMENTAL EDUCATION 119 69 1 11 ENVIRONMENTAL EDUCATION 139 69 1 1 ENVIRONMENTAL QUALITY 103 69 1 11 EQUATION 24 24 1 17 ESCAP 133 24 1 6 ESCAP 134 24 1 6 ESTABLISHMENT 63 24 1 6 ESTUARIES 39 69 1 2 ESTUARIES 53 24 1 3 ESTUARIES 53 69 1 2 ESTUARIES 61 69 1 2 ESTUARIES 62 24 1 2 ESTUARIES 62 69 1 1 ESTUARIES 75 69 1 4 ESTUARIES 76 69 1 5 ESTUARINE 61 24 1 2 ESTUARY 39 24 1 4 ET 29 24 1 5 ET 29 24 1 10 ET 81 24 1 22 ET 104 24 1 9 ET 120 24 1 11 ETHICS 145 69 1 2 ETUDE 31 24 1 8 EUROPE 37 24 1 17 EUROPE 97 69 1 6 EVALUATING 136 24 1 1 EVALUATION METHOD 127 69 1 1 EVALUATION METHOD 136 69 1 4 EVALUATION OF EDUCATION 87 69 1 16 EVALUATION OF EDUCATION 93 69 1 15 EVALUATION OF EDUCATION 96 69 1 9 EVALUATION OF EDUCATION 114 69 1 9 EVALUATION OF EDUCATION 136 69 1 1 EVAPORATION 17 69 1 5 EVAPORATION 21 24 1 5 EVAPORATION 21 24 1 25 EVAPORATION 21 69 1 1 EVAPORATION 33 24 1 5 EVAPOTRANSPIRATION 5 69 1 4 EVAPOTRANSPIRATION 19 24 1 6 EVAPOTRANSPIRATION 19 69 1 1 EVAPOTRANSPIRATION 20 24 1 5 EVAPOTRANSPIRATION 20 69 1 1 EVAPOTRANSPIRATION 24 24 1 6 EVAPOTRANSPIRATION 24 69 1 1 EVAPOTRANSPIRATION 33 69 1 3 EVERYDAY LIFE 95 69 1 4 EXAMINATIONS 88 69 1 8 EXCHANGE 6 24 1 6 EXCHANGE 7 24 1 5 EXCHANGE 8 24 1 4 EXISTENCE 145 24 1 4 EXPECTATIONS 86 24 1 9 EXPERIENCE 13 24 1 1 EXPERIENCE 105 24 1 9 EXPERIMENT 96 24 1 9 EXPERIMENTAL 131 24 1 16 EXPERIMENTS 87 69 1 19 EXPERTS 133 69 1 1 EXTRACTS 132 24 1 5 FACTORS 63 24 1 2 FAJARDO, JOSE CARLOS 122 70 1 1 FAMILY LIFE 95 69 1 11 FAUNA 53 69 1 4 FAUNA 66 24 1 2 FAUNA 66 69 1 2 FAURE, EDGAR 138 70 1 3 FEATURES 45 24 1 3 FEATURES 60 24 1 2 FEATURES 75 24 1 2 FEATURES 122 24 1 5 FELLOWSHIPS 125 69 1 3 FERTILIZERS 65 69 1 8 FERTILIZERS 71 69 1 8 FIELD 3 24 1 22 FIELD 4 24 1 14 FIELD 6 24 1 17 FIELD 7 24 1 7 FIELD 14 24 1 10 FIELDS 85 24 1 17 FILMS 91 69 1 13 FINAL 121 24 1 1 FINAL 125 24 1 1 FINAL 126 24 1 1 FIRST 88 24 1 8 FISH 65 24 1 7 FISH 65 69 1 2 FISH 66 24 1 1 FISH 66 69 1 1 FISH 67 24 1 1 FISHERIES 67 24 1 3 FISHERIES 67 69 1 1 FLOOD 42 24 1 7 FLOOD CONTROL 59 69 1 3 FLOODING 78 24 1 7 FLOODS 30 69 1 6 FLOODS 32 69 1 3 FLOODS 42 69 1 3 FLOODS 58 69 1 4 FLOODS 59 24 1 3 FLOODS 74 69 1 6 FLOODS 78 69 1 5 FLOODS 80 69 1 5 FLORA 30 69 1 5 FLORA 62 69 1 2 FLORAS 62 24 1 11 FLOW MEASUREMENT AND INSTRUMEN 16 69 1 4 FLOW MEASUREMENT AND INSTRUMEN 20 69 1 3 FLOW MEASUREMENT AND INSTRUMEN 32 69 1 6 FLOW MEASUREMENT AND INSTRUMEN 33 69 1 6 FOLKLORE 115 24 1 1 FOLKLORE 115 69 1 1 FOOD 65 69 1 5 FOOD 66 69 1 6 FOOD 70 69 1 8 FOOD 130 24 1 1 FOOD PRODUCTION 72 69 1 5 FOOD PRODUCTION 130 69 1 2 FOOD RESOURCES 67 69 1 5 FOOD RESOURCES 69 69 1 4 FOOD RESOURCES 130 69 1 6 FORCE 14 24 1 14 FOREIGN INVESTMENT 110 69 1 4 FOREIGN TRADE 116 69 1 7 FOREST 22 24 1 4 FORESTS 22 69 1 3 FORESTS 27 69 1 5 FORESTS 30 69 1 2 FORESTS 48 69 1 4 FORESTS 60 69 1 6 FORESTS 62 69 1 3 FORESTS 64 24 1 3 FORESTS 64 69 1 1 FORESTS 68 69 1 5 FORMATION 36 24 1 3 FORMATION 52 24 1 15 FORMERLY 138 24 1 10 FOSBERG, F. RAYMOND 55 70 1 1 FRADIER, GEORGES 103 70 1 1 FRANCE 101 69 1 3 FRANCE 151 69 1 3 FRANCO, C.M. 1 70 1 2 FRANEKER 98 24 1 11 FREEDOM 103 69 1 9 FRESH 52 24 1 4 FRESHWATER 32 69 1 4 FRESHWATER 43 69 1 2 FRESHWATER 49 69 1 4 FRESHWATER 52 69 1 7 FRESHWATER 60 69 1 4 FRESHWATER 61 69 1 3 FRESHWATER 66 69 1 4 FULLY 3 24 1 8 FUTURE 139 24 1 4 GALE, J. 5 70 1 1 GANDHIAN 142 24 1 5 GANGES 35 24 1 7 GANGES 78 24 1 20 GASEOUS 8 24 1 3 GAUGES 19 24 1 1 GAZ 81 24 1 9 GENERAL 60 24 1 1 GENERAL 75 24 1 1 GENERAL 138 24 1 12 GENESIS 41 24 1 2 GEODYNAMICS 35 69 1 1 GEODYNAMICS 36 69 1 2 GEODYNAMICS 37 69 1 1 GEOLOGICAL 55 24 1 4 GEOLOGY 55 69 1 2 GEOMORPHOLOGIE 31 24 1 5 GEOMORPHOLOGY 28 69 1 2 GEOMORPHOLOGY 29 69 1 4 GEOMORPHOLOGY 31 69 1 1 GEOMORPHOLOGY 34 69 1 1 GEOMORPHOLOGY 45 69 1 3 GEOMORPHOLOGY 50 69 1 6 GEOMORPHOLOGY 76 69 1 3 GEOMORPHOLOGY 77 69 1 1 GEOMORPHOLOGY 78 24 1 5 GEOMORPHOLOGY 78 69 1 1 GEOTHERMAL ENERGY 141 69 1 9 GERMANY FR 53 69 1 6 GHANI, Q. 64 70 1 1 GOODMAN, LOUIS WOLF 111 70 1 2 GOOR, G.A.W. VAN DE 71 70 1 1 GRADUAL 35 24 1 3 GRADUATES 86 69 1 3 GRASS 22 24 1 7 GREAT 45 24 1 6 GREATER 100 24 1 7 GREATER 150 24 1 7 GREECE 89 24 1 2 GREECE 89 69 1 5 GREECE 107 69 1 5 GRIEVE, B.J. 4 70 1 1 GROOT, A.J. DE 37 70 1 1 GROUND 57 24 1 1 GROUNDWATER 33 69 1 2 GROUNDWATER 57 69 1 1 GROUP DISCUSSIONS 93 69 1 11 GROWTH 56 24 1 9 GROWTH 147 24 1 2 GUIDANCE SERVICE 133 69 1 3 GUIDE 85 24 1 6 GUIDES 85 69 1 4 GUIDES 101 69 1 4 GUIDES 131 69 1 8 GUIDES 151 69 1 4 GUTTMANN, LUDWIG 92 70 1 2 HALLAK, JACQUES 104 70 1 1 HANDBOOK 140 24 1 9 HAVELOCK, MARY C. 85 70 1 2 HAVELOCK, RONALD G. 85 70 1 1 HEALTH 109 69 1 9 HEAT 15 24 1 2 HEAT 25 24 1 2 HEAT TRANSFER 25 69 1 5 HEEMST, H.D.J. VAN 24 70 1 2 HEMPHILL, JOHN KNOX 114 70 1 1 HERVIEU, J. 29 70 1 1 HIGH 90 24 1 9 HIGHER 86 24 1 1 HIGHER EDUCATION 79 69 1 6 HIGHER EDUCATION 86 69 1 4 HIGHER EDUCATION 97 69 1 2 HIGHER TECHNICAL PERSONNEL 119 69 1 14 HISTOIRE 98 24 1 5 HISTORIC MONUMENTS 94 69 1 1 HISTORICAL ANALYSIS 83 69 1 7 HISTORICAL ANALYSIS 116 69 1 5 HISTORY 144 69 1 3 HOLDINGS 106 24 1 5 HOLLANDER, JACK M. 109 70 1 1 HOLLERWOGER, F. 48 70 1 1 HOLT, MICHAEL 143 70 1 1 HOME 90 24 1 5 HOME 90 24 1 16 HOOK, SIDNEY 112 70 1 1 HORLAMUS, SEPP 113 70 1 1 HOUSE 138 24 1 16 HOUSING 95 69 1 12 HOUSING 103 69 1 5 HUDSON, J.P. 19 70 1 1 HUISMAN, L. 57 70 1 1 HUMAN 97 24 1 1 HUMAN ACTIVITIES EFFECTS 28 69 1 9 HUMAN ACTIVITIES EFFECTS 49 69 1 1 HUMAN ACTIVITIES EFFECTS 50 69 1 1 HUMAN ACTIVITIES EFFECTS 51 69 1 4 HUMAN ECOLOGY 139 69 1 5 HUMAN RESOURCES DEVELOPMENT 100 69 1 4 HUMAN RESOURCES DEVELOPMENT 150 69 1 4 HUMAN RIGHTS 97 69 1 1 HUMAN SETTLEMENT 103 69 1 4 HUMID 28 24 1 5 HUMID 41 24 1 15 HUMID 54 24 1 8 HUMID 64 24 1 11 HUMID 69 24 1 1 HUMID 71 24 1 9 HUMID ZONES 41 69 1 3 HUMID ZONES 50 69 1 8 HUMID ZONES 54 69 1 4 HUMID ZONES 60 69 1 2 HUMID ZONES 71 69 1 4 HUMIDITE 81 24 1 24 HUMIDITY 3 24 1 17 HUNGARY 98 69 1 5 HYDRATURE 14 24 1 7 HYDRAULIC 44 24 1 1 HYDRAULIC 49 24 1 4 HYDRAULIC ENGINEERING 44 69 1 1 HYDRAULIC ENGINEERING 47 69 1 1 HYDRAULIC ENGINEERING 49 69 1 2 HYDRODYNAMICS 36 69 1 7 HYDRODYNAMICS 57 69 1 9 HYDROGEOCHEMISTRY 33 69 1 1 HYDROGEOLOGY 36 69 1 1 HYDROGRAPHY 28 69 1 6 HYDROLOGICAL 47 24 1 1 HYDROLOGICAL CYCLE 57 69 1 8 HYDROLOGY 28 69 1 5 HYDROLOGY 34 69 1 6 HYDROLOGY 36 24 1 1 HYDROLOGY 58 24 1 2 HYDROLOGY 58 69 1 1 HYDROLOGY 74 69 1 3 HYDROLOGY 76 69 1 6 HYDROLOGY 77 69 1 2 HYDROMETEOROLOGICAL 80 24 1 1 HYDROMETEOROLOGY 80 69 1 1 HYGROMETER 4 24 1 3 HYGROMETERS 4 69 1 1 HYGROMETERS 11 69 1 4 IBE 106 24 1 8 IDEAL 102 24 1 3 IMPLICATIONS 28 24 1 11 IMPORTANT 70 24 1 2 IMPROVEMENT 49 24 1 5 IN-SERVICE TEACHER TRAINING 84 69 1 8 IN-SERVICE TEACHER TRAINING 96 69 1 8 INCLUDING 44 24 1 6 INCOME 95 69 1 17 INCOME DISTRIBUTION 147 69 1 7 INCOMUNICACAO 108 24 1 2 INDEX 10 24 1 14 INDEX 14 24 1 17 INDIA 44 69 1 6 INDIA 58 69 1 6 INDIA 68 69 1 7 INDIA 78 69 1 11 INDIA 80 24 1 10 INDIA 80 69 1 9 INDIA 84 24 1 4 INDIA 84 69 1 3 INDIA 96 69 1 5 INDIA 142 69 1 5 INDIAN 96 24 1 6 INDIRECT 26 24 1 1 INDIVIDUAL 1 24 1 8 INDONESIA 41 69 1 8 INDONESIA 48 69 1 7 INDONESIA 94 69 1 3 INDUSTRIAL PRODUCTS 110 69 1 6 INDUSTRIALIZATION 100 69 1 9 INDUSTRIALIZATION 118 69 1 6 INDUSTRIALIZATION 150 69 1 9 INDUSTRIES 73 69 1 4 INFLUENCE 2 24 1 10 INFLUENCE 49 24 1 2 INFLUENCE 50 24 1 1 INFLUENCE 64 24 1 1 INFORMATION PROCESSING AUTOMAT 137 69 1 4 INFORMATION SERVICES 137 69 1 7 INFORMATION SOURCES 113 69 1 12 INFORMATION SYSTEMS 114 69 1 10 INFORMATION USERS 137 69 1 8 INFORMATION/LIBRARY FINANCING 137 69 1 9 INFORMATIQUE 137 24 1 3 INFRA 6 24 1 8 INNOVATION 87 24 1 1 INNOVATION 92 24 1 5 INNOVATION 110 24 1 11 INNOVATION BEHAVIOUR 83 69 1 6 INSTRUCTIONAL 96 24 1 7 INTEGRATED CURRICULUM 87 69 1 12 INTERACTION 43 24 1 1 INTERACTION 144 24 1 2 INTERDISCIPLINARY APPROACH 121 69 1 5 INTERGOVERNMENTAL ORGANIZATION 116 69 1 1 INTERNAL 10 24 1 17 INTERNAL 14 24 1 4 INTERNATIONAL 116 24 1 9 INTERNATIONAL CONFLICT 107 69 1 2 INTERNATIONAL CONFLICT 116 69 1 3 INTERNATIONAL COOPERATION 102 69 1 4 INTERNATIONAL COOPERATION 132 69 1 2 INTERNATIONAL COOPERATION 147 69 1 15 INTERNATIONAL LAW 116 69 1 6 INTERNATIONAL ORGANIZATIONS 132 69 1 5 INTERNATIONAL RELATIONS 116 69 1 4 INTERNATIONAL STANDARDS 131 69 1 7 INTERNATIONAL TRADE 111 69 1 5 INTERNATIONALE 98 24 1 2 INVERTEBRATA 68 69 1 1 INVOLVED 63 24 1 3 IRAN (ISLAMIC REPUBLIC) 133 69 1 5 IRAN (ISLAMIC REPUBLIC) 134 69 1 6 IRRAWADDY 74 24 1 6 IRRAWADDY 78 24 1 18 IRRIGATION 9 24 1 14 IRRIGATION 47 69 1 3 IRRIGATION 49 69 1 6 IRRIGATION 71 69 1 7 ISLAM, M. AMIRUL 38 70 1 1 ISRAEL 22 69 1 7 IT 52 24 1 10 ITALY 32 69 1 7 JACKMAN, MARY ELIZABETH 86 70 1 4 JAVA 48 24 1 8 JOBORU, MAGDA 138 70 1 1 JOURNALISM 113 69 1 2 JOURNALISTS 91 69 1 19 JOURNALISTS 113 69 1 6 JUSQU 120 24 1 13 KARAMAZOV 129 24 1 13 KARENINA 129 24 1 10 KARIM, A.Q.M.B. 40 70 1 1 KENYA 148 69 1 3 KHAN, F. KARIM 73 70 1 2 KHUDA, M. QUDRAT 69 70 1 1 KIDD, JAMES ROBBINS 84 70 1 2 KLAUSING, O. 21 70 1 1 KLEINFELD, JUDITH S. 90 70 1 1 KRAMER, P.J. 11 70 1 1 KREEB, K. 14 70 1 1 L 31 24 1 7 L 81 24 1 6 L 98 24 1 4 L 124 24 1 4 L 124 24 1 23 L 137 24 1 1 LA 30 24 1 3 LA 31 24 1 1 LA 31 24 1 4 LA 31 24 1 10 LA 33 24 1 1 LA 81 24 1 3 LA 104 24 1 1 LA 104 24 1 12 LA 120 24 1 1 LA 123 24 1 4 LA 124 24 1 12 LA 124 24 1 15 LA 124 24 1 18 LA 129 24 1 15 LABORATORY 7 24 1 9 LABOUR 86 24 1 5 LABOUR MARKET 86 69 1 6 LABOUR MIGRATION 147 69 1 9 LABOUR POLICY 147 69 1 1 LABOUR RELATIONS 83 69 1 5 LABOUR RELATIONS 95 69 1 23 LAND 73 24 1 3 LAND RECLAMATION 49 69 1 8 LAND RECLAMATION 67 69 1 2 LAND USE 73 69 1 1 LAND USE 100 69 1 10 LAND USE 150 69 1 10 LANGE, O.L. 15 70 1 1 LANGUAGE BEHAVIOUR 82 69 1 5 LANGUAGE OF INSTRUCTION 127 69 1 13 LATIN 105 24 1 7 LATIN AMERICA 97 69 1 5 LATIN AMERICA 105 69 1 3 LAW 116 24 1 10 LE 29 24 1 15 LEADERSHIP 127 69 1 12 LEAF 12 24 1 1 LEARNING 117 24 1 5 LEARNING 149 24 1 8 LEARNING PROCESSES 87 69 1 21 LEARNING PROCESSES 93 69 1 3 LEARNING PROCESSES 117 69 1 5 LEARNING PROCESSES 143 69 1 3 LEAST DEVELOPED COUNTRIES 119 69 1 10 LEAVES 10 24 1 21 LEAVES 10 69 1 3 LEBLOND, C. 81 70 1 1 LECTURES 93 69 1 10 LEGISLATION 101 69 1 2 LEGISLATION 151 69 1 2 LEIDEN 98 24 1 9 LEKOVIC, ZDRAVKO 91 70 1 1 LEROY, GEORGES 110 70 1 1 LES 29 24 1 2 LES 30 24 1 6 LES 32 24 1 5 LES 33 24 1 10 LES 81 24 1 12 LES 120 24 1 7 LES 123 24 1 10 LES 124 24 1 8 LEVITT, J. 16 70 1 1 LIBERATION MOVEMENTS 125 69 1 6 LIBERATION MOVEMENTS 126 69 1 1 LIETH, H. 26 70 1 1 LIFE 103 24 1 5 LIFE 122 24 1 7 LIFE-LONG EDUCATION 84 69 1 12 LIFE-LONG EDUCATION 88 69 1 11 LIFE-LONG EDUCATION 121 69 1 2 LIFE-LONG EDUCATION 122 69 1 1 LIFE-LONG EDUCATION 146 69 1 3 LIFELONG 146 24 1 4 LIKELY 119 24 1 14 LIST 106 24 1 1 LITERACY PROGRAMMES 84 69 1 4 LITERARY CRITICISM 129 69 1 2 LITERATURE 22 24 1 14 LITERATURE 79 69 1 9 LIVING CONDITIONS 74 69 1 8 LIVING CONDITIONS 95 69 1 8 LONG 90 24 1 2 LONG 122 24 1 8 LOSS 4 24 1 9 LUMIERE 124 24 1 13 LYSIMETER 21 24 1 18 M 113 24 1 5 MADAGASCAR 29 24 1 22 MADAGASCAR 29 69 1 8 MAGALHAES, A.C. 1 70 1 1 MAHEU 138 24 1 9 MAIN 45 24 1 1 MAKING 83 24 1 6 MAKKINK, G.F. 24 70 1 1 MAN 50 24 1 3 MANAGEMENT 149 24 1 19 MANAGEMENT INFORMATION SYSTEMS 137 69 1 3 MANAGEMENT OPERATIONS 83 69 1 2 MANGANESE 37 24 1 5 MANGANESE 37 69 1 3 MANGOKY 29 24 1 18 MANGROVE 63 24 1 5 MANGROVES 68 24 1 7 MANKIND 94 24 1 6 MANPOWER NEEDS 86 69 1 8 MANPOWER NEEDS 147 69 1 8 MANUEL 101 24 1 1 MANUEL 151 24 1 1 MANUFACTURING INDUSTRIES 141 69 1 13 MAQUIS 22 24 1 5 MARJORAM, D.T.E. 143 70 1 2 MARKET 86 24 1 6 MARRAKECH 124 24 1 21 MARSHES 61 69 1 4 MARSHES 62 69 1 5 MASS 113 24 1 1 MASS CULTURE 108 69 1 3 MASS MEDIA 84 69 1 5 MASS MEDIA 91 69 1 2 MASS MEDIA 108 69 1 1 MASS MEDIA 113 69 1 1 MASS MEDIA 135 69 1 3 MASTER 139 24 1 6 MATHEMATICAL 44 24 1 7 MATHEMATICAL LOGIC 143 69 1 4 MATHEMATICAL MODELS 24 69 1 4 MATHEMATICAL MODELS 44 69 1 2 MATHEMATICAL MODELS 143 69 1 5 MATHEMATICS 143 24 1 1 MATHEMATICS 143 69 1 1 MATTER 26 24 1 7 MATTER 27 24 1 4 MAY, WILLIAM H. 82 70 1 2 MCQUAIL, DENIS 135 70 1 1 MEANING 82 24 1 6 MEANS 149 24 1 14 MEASUREMENT 1 24 1 4 MEASUREMENT 11 24 1 1 MEASUREMENT 16 24 1 2 MEASUREMENT 17 24 1 1 MEASUREMENT 18 24 1 16 MEASUREMENT 26 24 1 4 MEASUREMENT 27 24 1 1 MEASUREMENT 131 24 1 2 MEASUREMENT AND INSTRUMENTS 1 69 1 3 MEASUREMENT AND INSTRUMENTS 3 69 1 5 MEASUREMENT AND INSTRUMENTS 5 69 1 5 MEASUREMENT AND INSTRUMENTS 8 69 1 3 MEASUREMENT AND INSTRUMENTS 12 69 1 3 MEASUREMENT AND INSTRUMENTS 13 69 1 4 MEASUREMENT AND INSTRUMENTS 14 69 1 5 MEASUREMENT AND INSTRUMENTS 15 69 1 3 MEASUREMENT AND INSTRUMENTS 19 69 1 3 MEASUREMENT AND INSTRUMENTS 20 69 1 5 MEASUREMENT AND INSTRUMENTS 21 69 1 5 MEASUREMENT AND INSTRUMENTS 25 69 1 6 MEASUREMENT AND INSTRUMENTS 81 69 1 3 MEASUREMENTS 6 24 1 1 MEASUREMENTS 21 24 1 22 MEASUREMENTS 25 24 1 13 MEASURING 4 24 1 6 MEASURING 9 24 1 7 MEASURING 13 24 1 7 MEASURING 14 24 1 12 MEDIA 113 24 1 2 MEDICINAL PLANTS 69 69 1 3 MEETING 119 24 1 20 MEETINGS 119 24 1 9 MELO, JOSE MARQUES DE 108 70 1 1 MEMORIAL 138 24 1 5 MESURE 81 24 1 4 METABOLISM 27 69 1 6 METEOROLOGICAL 42 24 1 2 METEOROLOGY 42 69 1 1 METHOD 12 24 1 3 METHOD 20 24 1 2 METHOD 112 24 1 3 METHODOLOGIE 104 24 1 10 METHODOLOGY 46 24 1 1 METHODOLOGY 46 69 1 5 METHODOLOGY 86 69 1 9 METHODOLOGY 104 69 1 3 METHODOLOGY 112 69 1 4 METHODS 13 24 1 5 METHODS 25 24 1 11 METHODS 26 24 1 2 METHODS 129 24 1 22 MICHAELI, A. 22 70 1 2 MICHIGAN 139 24 1 1 MICROBIOLOGY 54 24 1 5 MICROBIOLOGY 54 69 1 2 MICROCLIMATE 64 24 1 8 MICROCLIMATIC 3 24 1 21 MICROPOLITAN 100 24 1 1 MICROPOLITAN 150 24 1 1 MIDDLEMARCH 129 24 1 8 MIKHAILOV, V.N. 36 70 1 1 MIKHAILOV, V.N. 43 70 1 2 MILITARY ENGINEERING 140 69 1 7 MINERAL DEPOSITS 29 69 1 2 MINERAL RESOURCES 39 69 1 6 MINERAL RESOURCES 64 69 1 6 MINERALOGIQUES 29 24 1 6 MIRON, MURRAY S. 82 70 1 3 MISE 31 24 1 11 MISE 104 24 1 2 MODEL 24 24 1 2 MODEL 92 24 1 3 MODELS 44 24 1 3 MODELS 44 24 1 8 MODIFICATIONS 10 24 1 1 MOISTURE 3 69 1 2 MOISTURE 4 69 1 3 MOISTURE 10 69 1 2 MOISTURE 12 69 1 2 MOISTURE 18 69 1 3 MOISTURE 21 24 1 21 MOISTURE 81 69 1 4 MOLECULAR BIOLOGY 99 69 1 4 MOLLUSCA 68 24 1 5 MONGOLIA 113 69 1 5 MONOLITHS 20 24 1 8 MONUMENT 94 24 1 4 MOORMANN, F.R. 41 70 1 1 MORPHOLOGY 41 24 1 1 MORPHOLOGY 41 69 1 2 MORPHOLOGY (LINGUISTICS) 82 69 1 4 MOUTH 36 24 1 6 MR 122 24 1 13 MUCHEKE 95 24 1 1 MUD 37 24 1 1 MUELLER-DOMBOIS, D. 52 70 1 2 MULTINATIONAL 110 24 1 1 MULTINATIONAL 111 24 1 1 MULTIRACIAL SOCIETIES 95 69 1 9 NAGARAJA, V.N. 44 70 1 1 NAGARAJA, V.N. 80 70 1 1 NAMIBIA 125 69 1 7 NAQVI, SIBTE NABI 42 70 1 1 NATIONAL 130 24 1 6 NATIONAL DEVELOPMENT 86 69 1 24 NATIONAL DEVELOPMENT 125 69 1 5 NATIONAL DEVELOPMENT 127 69 1 15 NATIONAL DEVELOPMENT 130 69 1 4 NATIONAL PLANNING 139 69 1 4 NATIONAL POLICY 109 69 1 11 NATIONALISM 95 69 1 20 NATIONS 116 24 1 3 NATURAL 77 24 1 13 NATURAL 145 24 1 3 NEED 73 24 1 1 NEED 136 24 1 5 NEEDS 147 24 1 5 NETHERLANDS 98 69 1 4 NEW 9 24 1 2 NEW 114 24 1 4 NEWLY-INDEPENDENT STATES 125 69 1 8 NEWS AGENCIES 91 69 1 12 NEWS AGENCIES 113 69 1 9 NEWSPAPERS 113 69 1 8 NIGER 31 69 1 9 NIGER 77 69 1 9 NO 108 24 1 3 NON-VIOLENCE 142 69 1 1 NORTH AMERICA 97 69 1 4 NOTES 10 24 1 8 NOVEL 129 24 1 4 NOVELS 129 69 1 1 NUCLEAR 140 24 1 6 NUCLEAR WEAPONS 140 69 1 1 NUTRIENT 40 24 1 1 NUTRIENTS 40 69 1 5 NUTRITION 69 69 1 2 NUTRITION 130 24 1 3 NUTRITION 130 69 1 3 OBJECTIVES 149 24 1 13 OBLIGATOIRE 124 24 1 25 OCCUPANT 30 24 1 5 OCCUPATIONAL QUALIFICATIONS 86 69 1 13 OCCURENCE 41 24 1 4 OCEAN CURRENTS 77 69 1 7 OCEAN CURRENTS 80 69 1 7 OCEANIA 97 69 1 8 OHYA, MASAHIKO 78 70 1 1 OIAPOQUE 62 24 1 7 OKATCHA, F.M.M.O. 148 70 1 1 ONE 147 24 1 7 ONTARIO 117 24 1 7 OPEN 88 24 1 1 OPENING 9 24 1 9 OPERATIONS 35 24 1 15 ORGANES 81 24 1 13 ORGANIC MATTER 26 69 1 1 ORGANIC MATTER 27 69 1 1 ORGANIC MATTER 55 69 1 7 ORGANIC MATTER 65 69 1 6 ORGANIZATION 149 24 1 17 ORGANIZATIONAL 83 24 1 4 OSGOOD, CHARLES EGERTON 82 70 1 1 OTHER 24 24 1 11 OTHER 76 24 1 8 OTHER 85 24 1 16 OTRAS 118 24 1 3 OUEST 29 24 1 20 PACIFIC ISLANDS 134 69 1 7 PAKISTAN 38 24 1 4 PAKISTAN 40 24 1 9 PAKISTAN 42 24 1 11 PAKISTAN 59 24 1 6 PAKISTAN 60 24 1 5 PAKISTAN 70 24 1 9 PAKISTAN 72 24 1 4 PAKISTAN 73 24 1 8 PAR 33 24 1 4 PAR 81 24 1 11 PARASITES 70 24 1 6 PARASITES 70 69 1 2 PART 117 24 1 3 PART-TIME COURSES 117 69 1 2 PATEL, A.M. 72 70 1 1 PAYS 120 24 1 8 PAYS 123 24 1 11 PAYS 124 24 1 9 PECS 98 24 1 10 PERDU 129 24 1 19 PERFORMANCE 86 24 1 11 PERIODICALS 106 24 1 3 PERRY, WALTER 88 70 1 1 PERSONAL 88 24 1 4 PERSONAL 148 24 1 1 PERSONNEL MANAGEMENT 83 69 1 4 PERSPECTIVES 146 24 1 2 PERU 75 69 1 7 PERU 92 69 1 3 PERU 105 69 1 9 PERUVIAN 92 24 1 2 PESTS 68 69 1 4 PESTS 70 24 1 4 PESTS 70 69 1 1 PETTIGREW, ANDREW M. 83 70 1 1 PHENOMENOLOGY 145 24 1 1 PHILIPPINES 58 69 1 11 PHILOSOPHICAL 129 24 1 5 PHILOSOPHY 99 24 1 4 PHILOSOPHY 99 69 1 2 PHILOSOPHY 112 24 1 5 PHILOSOPHY 112 69 1 2 PHILOSOPHY 129 24 1 1 PHILOSOPHY 142 24 1 6 PHILOSOPHY 142 69 1 3 PHILOSOPHY 144 24 1 6 PHILOSOPHY 144 69 1 2 PHILOSOPHY 145 69 1 1 PHOTOGRAPHS 34 24 1 5 PHOTOSYNTHESIS 7 69 1 4 PHOTOSYNTHESIS 12 69 1 4 PHOTOSYNTHESIS 63 69 1 4 PHOTOSYNTHESIS 65 69 1 7 PHYA 37 24 1 22 PHYA 78 24 1 17 PHYSICAL EDUCATION 119 69 1 16 PHYSICAL PROPERTIES 14 69 1 4 PHYSICAL PROPERTIES 29 69 1 5 PHYSICAL PROPERTIES 39 69 1 3 PHYSIOLOGY 8 24 1 5 PIED 81 24 1 16 PILOT 105 24 1 12 PILOT PROJECTS 96 69 1 2 PILOT PROJECTS 104 69 1 7 PILOT PROJECTS 105 69 1 1 PINE 18 24 1 6 PIRES, J. MURCA 62 70 1 1 PLACE 31 24 1 2 PLACE 104 24 1 4 PLAINS 42 24 1 8 PLAINS 78 24 1 10 PLAN 139 24 1 7 PLANKTON 65 24 1 1 PLANKTON 65 69 1 1 PLANNING 96 24 1 1 PLANT 2 24 1 6 PLANT 3 24 1 6 PLANT 5 24 1 17 PLANT 6 24 1 3 PLANT 8 24 1 9 PLANT 21 24 1 8 PLANT 25 24 1 9 PLANT 27 24 1 8 PLANT ECOLOGY 15 69 1 2 PLANT ECOLOGY 21 69 1 4 PLANT ECOLOGY 22 69 1 4 PLANT ECOLOGY 26 69 1 5 PLANT EVAPOTRANSPIRATION 2 69 1 1 PLANT PHOTOSYNTHESIS 6 69 1 1 PLANT PHYSIOLOGY 1 69 1 1 PLANT PHYSIOLOGY 3 69 1 1 PLANT PHYSIOLOGY 5 69 1 1 PLANT PHYSIOLOGY 7 69 1 1 PLANT PHYSIOLOGY 8 69 1 1 PLANT PHYSIOLOGY 9 69 1 1 PLANT PHYSIOLOGY 10 69 1 1 PLANT PHYSIOLOGY 11 69 1 1 PLANT PHYSIOLOGY 12 69 1 1 PLANT PHYSIOLOGY 13 69 1 1 PLANT PHYSIOLOGY 14 69 1 2 PLANT PHYSIOLOGY 16 69 1 2 PLANT PHYSIOLOGY 17 69 1 4 PLANT PHYSIOLOGY 18 69 1 1 PLANT PHYSIOLOGY 21 69 1 3 PLANT PHYSIOLOGY 25 69 1 4 PLANT PHYSIOLOGY 26 69 1 2 PLANT PHYSIOLOGY 27 69 1 2 PLANT PHYSIOLOGY 63 69 1 3 PLANT PHYSIOLOGY 81 69 1 1 PLANT TRANSPIRATION 1 69 1 2 PLANT TRANSPIRATION 4 69 1 2 PLANT TRANSPIRATION 5 69 1 3 PLANT TRANSPIRATION 8 69 1 2 PLANT TRANSPIRATION 19 69 1 2 PLANT TRANSPIRATION 24 69 1 3 PLANTS 1 24 1 9 PLANTS 4 24 1 11 PLANTS 11 24 1 6 PLANTS 13 24 1 11 PLANTS 15 24 1 5 PLANTS 17 24 1 7 PLANTS 55 69 1 6 PLANTS 70 69 1 5 POCZTAR, JERRY 149 70 1 1 POINT 18 24 1 13 POLICIES 91 24 1 2 POLICY 79 24 1 2 POLITICAL BEHAVIOUR 95 69 1 3 POLITICAL CONFLICT 107 69 1 1 POLITICAL ECONOMY 111 69 1 6 POLITICAL PARTICIPATION 79 69 1 7 POLITICAL PARTIES 95 69 1 21 POLITICAL SYSTEMS 91 69 1 6 POLITICAL THOUGHT 142 69 1 4 POLITICS 83 24 1 2 POLITICS 95 24 1 5 POLITIQUES 104 24 1 6 POLJAKOFF-MAYBER, A. 5 70 1 2 POLLUTION 109 69 1 7 POLSTER, H. 8 70 1 1 PONENCIAS 118 24 1 1 POPULATION 72 24 1 1 POPULATION DENSITY 72 69 1 1 POPULATION DENSITY 74 69 1 5 POPULATION DYNAMICS 72 69 1 2 POPULATION DYNAMICS 143 69 1 8 POROMETER 9 24 1 5 PORTUGAL 98 24 1 8 PORTUGAL 98 69 1 3 POST-GRADUATE COURSES 88 69 1 10 POTENTIAL 13 24 1 9 POTENTIAL 21 24 1 24 POUR 81 24 1 2 POVERTY 100 69 1 7 POVERTY 147 69 1 6 POVERTY 150 69 1 7 POWER INDUSTRY 109 69 1 5 PRACTICE 100 24 1 5 PRACTICE 131 24 1 10 PRACTICE 150 24 1 5 PRATIQUE 101 24 1 2 PRATIQUE 151 24 1 2 PRESENTED 54 24 1 2 PRESENTED 122 24 1 11 PRESERVATION OF CULTURAL PROPE 94 69 1 9 PRESERVATION OF MONUMENTS 94 69 1 4 PRESS 91 69 1 10 PRESSURE 11 69 1 3 PRESSURE 13 69 1 3 PRESSURE 14 69 1 3 PREVAILING 77 24 1 15 PRIMAIRE 124 24 1 6 PRIMARY EDUCATION 104 69 1 11 PRIMARY EDUCATION 124 69 1 3 PRIMARY SCHOOLS 104 69 1 4 PROBLEM 147 24 1 9 PROBLEMES 32 24 1 1 PROBLEMS 28 24 1 2 PROBLEMS 42 24 1 3 PROBLEMS 47 24 1 5 PROBLEMS 54 24 1 1 PROBLEMS 80 24 1 4 PROBLEMS 99 24 1 10 PROCEEDINGS 28 24 1 12 PRODUCT 110 24 1 2 PRODUCT 110 24 1 10 PRODUCTION 26 24 1 8 PRODUCTION 27 24 1 5 PRODUCTION 87 69 1 17 PRODUCTION 149 69 1 13 PRODUCTIVITY 65 24 1 2 PROFESSIONAL ASSOCIATIONS 91 69 1 18 PROFESSIONAL ASSOCIATIONS 113 69 1 7 PROFESSIONAL TRAINING 85 69 1 7 PROGRAMMED 149 24 1 7 PROGRAMMED INSTRUCTION 93 69 1 13 PROGRAMMED INSTRUCTION 149 69 1 2 PROGRAMMED MATERIALS 149 69 1 14 PROGRAMMES 136 24 1 3 PROGRAMS 85 24 1 12 PROGRESS 48 24 1 2 PROJECT 149 24 1 4 PROJECTIONS 120 24 1 12 PROJECTS 105 24 1 13 PROPOSED 131 24 1 8 PROWSE, G.A. 65 70 1 1 PSYCHOANALYSIS 112 24 1 1 PSYCHOANALYSIS 112 69 1 1 PSYCHOLINGUISTICS 82 69 1 1 PUBLIC 90 24 1 8 PUBLIC EDUCATION 114 69 1 2 PUBLIC HEALTH 70 69 1 7 PUBLIC OPINION 91 69 1 4 PUBLIC OPINION 108 69 1 7 QUALITY 103 24 1 3 QUALITY OF LIFE 103 69 1 1 QUANTITATIF 124 24 1 2 QUANTITATIFS 123 24 1 2 QUESTIONNAIRES 105 69 1 14 QURESHI, M. RAHIMULLAH 66 70 1 1 R 45 24 1 16 RACE 95 24 1 2 RACE RELATIONS 95 69 1 1 RACIAL SEGREGATION 95 69 1 10 RADIO 91 69 1 11 RADIO 113 69 1 10 RAIN 22 69 1 5 RAIN 38 69 1 5 RAIN 42 69 1 6 RAIN 75 69 1 3 RAIN 78 69 1 4 RAIN 80 69 1 4 RATES 19 24 1 7 READINGS 84 24 1 8 RECENT 119 24 1 6 RECENTES 30 24 1 8 RECENTS 29 24 1 9 RECHERCHE 129 24 1 16 RECOMMENDATIONS 119 24 1 4 RECOMMENDATIONS 132 24 1 3 RED 6 24 1 9 REDUCTION 99 24 1 7 REFERENCE 54 24 1 11 REFERENCE 71 24 1 4 REFORM 92 24 1 7 REFRACTIVE 14 24 1 16 REGARD 8 24 1 7 REGIME 56 24 1 5 REGIME OF WATERS 36 69 1 5 REGIME OF WATERS 56 69 1 1 REGIONAL COOPERATION 133 69 1 2 REGIONAL COOPERATION 134 69 1 9 REGIONAL PLANNING 100 69 1 1 REGIONAL PLANNING 150 69 1 1 REGIONS 22 24 1 12 REGIONS 76 24 1 10 REGULATION 3 24 1 10 RELATED 99 24 1 9 RELATING 21 24 1 17 RELATION 65 24 1 4 RELATIVE 3 24 1 16 RELATIVE 10 24 1 4 RELATIVE 81 24 1 25 RELEVANCE 119 24 1 11 RENE 138 24 1 8 RENEWABLE 141 24 1 5 RENEWABLE RESOURCES 141 69 1 1 RENEWAL 114 24 1 8 REPORT 121 24 1 2 REPORT 125 24 1 2 REPORT 126 24 1 2 REPORT 127 24 1 1 REPUBLIC 79 24 1 6 REPUBLIC 107 24 1 4 RESEARCH 5 24 1 5 RESEARCH 131 24 1 14 RESEARCH AND DEVELOPMENT 105 69 1 11 RESEARCH AND DEVELOPMENT 131 69 1 3 RESEARCH AND DEVELOPMENT 141 69 1 14 RESEARCH CENTRES 105 69 1 17 RESEARCH CENTRES 141 69 1 15 RESEARCH PRIORITIES 130 69 1 7 RESEARCH WORKERS 105 69 1 18 RESISTANCE 15 24 1 3 RESISTANCE 16 24 1 5 RESISTANCE 17 24 1 4 RESOLUTION 124 24 1 16 RESOLUTIONS 119 24 1 2 RESOURCE CONSERVATION 109 69 1 4 RESOURCES 141 24 1 6 RESOURCES 149 24 1 16 RESPONSE 136 24 1 8 RESTORATION 94 69 1 8 REVIEW 22 24 1 15 REVIEW 109 24 1 2 RHODESIA 116 24 1 5 RHODESIAN 95 24 1 8 RICE 71 24 1 6 RIEPE, DALE MAURICE 145 70 1 1 RIGHT TO INFORMATION 91 69 1 16 RIGHTS 97 24 1 2 RIVER 36 24 1 5 RIVER 37 24 1 23 RIVER 43 24 1 3 RIVER 48 24 1 5 RIVER 56 24 1 4 RIVER 74 24 1 7 RIVER BASINS 48 69 1 1 RIVER BASINS 49 69 1 5 RIVER BASINS 62 69 1 4 RIVER CONTROL 59 69 1 4 RIVERS 43 69 1 1 RIVERS 53 69 1 5 RIVERS 58 69 1 2 RIVERS 62 24 1 8 RIVERS 67 69 1 3 RIVERS 74 69 1 1 RIVERS 75 69 1 2 ROLE 18 24 1 2 ROLE 104 24 1 8 ROLE OF EDUCATION 119 69 1 17 ROLE OF EDUCATION 121 69 1 4 ROLE OF EDUCATION 128 69 1 7 ROONWAL, M.L. 68 70 1 1 ROSENAU, FRED S. 114 70 1 2 RUN-OFF 22 69 1 6 RURAL 100 24 1 8 RURAL 141 24 1 3 RURAL 150 24 1 8 RURAL AREAS 90 69 1 3 RURAL AREAS 104 69 1 15 RURAL DEVELOPMENT 100 69 1 2 RURAL DEVELOPMENT 119 69 1 18 RURAL DEVELOPMENT 141 69 1 4 RURAL DEVELOPMENT 150 69 1 2 RURAL POPULATION 72 69 1 3 RURAL YOUTH 90 69 1 8 S 45 24 1 14 S 45 24 1 15 S 50 24 1 4 S 139 24 1 2 SA, ARTUR MOREIRA DE 98 70 1 2 SAFETY 109 69 1 10 SAINT ANTONIN, CLAUDE 137 70 1 1 SALINE 52 24 1 7 SALINE WATER 32 69 1 5 SALINE WATER 57 69 1 6 SALINE WATER 60 69 1 5 SALINE WATER 61 69 1 5 SALINE WATER 63 69 1 2 SALINITY 33 69 1 5 SALINITY 38 69 1 6 SALINITY 40 69 1 3 SALINITY 44 69 1 5 SALINITY 52 69 1 5 SALINITY 53 69 1 3 SALINITY 63 69 1 1 SALINITY 80 69 1 8 SAMMAK, A. 123 70 1 1 SANCTIONS 116 69 1 8 SANTEMA, P. 77 70 1 1 SANYAL, BIKAS C. 86 70 1 1 SATELLITE 96 24 1 3 SATELLITE BROADCASTING 96 69 1 3 SATURATION 12 24 1 7 SCALE 44 24 1 2 SCHMID, MAURICE 30 70 1 1 SCHOELLER, MARC 33 70 1 1 SCHOLARSHIPS 134 69 1 10 SCHOOL LABORATORIES 87 69 1 20 SCHOOL MAPPING 104 69 1 1 SCHOOL-AGE POPULATION 120 69 1 7 SCHOOL-AGE POPULATION 124 69 1 8 SCHOOLS 90 24 1 10 SCIENCE 87 24 1 3 SCIENCE 105 24 1 3 SCIENCE 144 24 1 4 SCIENCE 144 69 1 1 SCIENCE AND DEVELOPMENT 87 69 1 7 SCIENCE AND DEVELOPMENT 119 69 1 5 SCIENCE AND TECHNOLOGY 87 69 1 5 SCIENCE AND TECHNOLOGY 109 69 1 6 SCIENCE AND TECHNOLOGY 118 69 1 1 SCIENCE AND TECHNOLOGY 119 69 1 4 SCIENCE AND TECHNOLOGY 131 69 1 2 SCIENCE EDUCATION 87 69 1 2 SCIENCE FINANCING 105 69 1 19 SCIENCE POLICY 118 69 1 2 SCIENCE STATISTICS 105 69 1 2 SCIENCE STATISTICS 131 69 1 4 SCIENTIFIC 28 24 1 1 SCIENTIFIC 112 24 1 2 SCIENTIFIC 131 24 1 4 SCIENTIFIC ACTIVITIES 105 69 1 6 SCIENTIFIC DEVELOPMENT 118 69 1 4 SCIENTIFIC INSTITUTIONS 118 69 1 9 SCIENTIFIC PERSONNEL 105 69 1 12 SCIENTIFIC PROGRAMMES 102 69 1 3 SCIENTIFIC PROGRESS 103 69 1 3 SCIENTIFIC RESEARCH 105 69 1 20 SCIENTIFIC RESEARCH 118 69 1 5 SCOLAIRE 104 24 1 14 SCOLAIRES 120 24 1 5 SCOLARISATION 123 24 1 5 SCULPTURE 94 69 1 7 SEA 43 24 1 5 SEA LEVEL 42 69 1 4 SEA WATER 43 69 1 3 SEA WATER 66 69 1 3 SECOND 123 24 1 7 SECONDARY EDUCATION 86 69 1 15 SECONDARY EDUCATION 90 69 1 1 SECONDARY EDUCATION 123 69 1 3 SECONDARY SCHOOL CURRICULUM 86 69 1 16 SECONDARY SCHOOL CURRICULUM 87 69 1 4 SECONDARY SCHOOLS 104 69 1 5 SEDIMENT 77 69 1 5 SEDIMENT TRANSPORT 37 69 1 2 SEDIMENT TRANSPORT 56 69 1 2 SEDIMENT TRANSPORT 58 69 1 5 SEDIMENTATION 28 69 1 3 SEDIMENTATION 29 69 1 1 SEDIMENTATION 32 24 1 3 SEDIMENTATION 32 69 1 1 SEDIMENTATION 35 69 1 3 SEDIMENTATION 36 69 1 4 SEDIMENTATION 38 69 1 4 SEDIMENTATION 39 69 1 5 SEDIMENTATION 40 69 1 6 SEDIMENTATION 48 69 1 3 SEDIMENTATION 52 69 1 6 SEDIMENTATION 55 69 1 5 SEDIMENTS 29 24 1 8 SELECTED 119 24 1 1 SELF 114 24 1 7 SELF-TEACHING METHODS 93 69 1 12 SEMANTICS 82 69 1 2 SEMI 22 24 1 10 SENEGAL 31 69 1 8 SEPTEMBER 106 24 1 13 SERVICES 133 24 1 2 SEX 120 69 1 10 SEX 123 69 1 7 SHACHORI, A.Y. 22 70 1 1 SHIFTING 35 24 1 4 SHIPWORMS 68 24 1 4 SHUI 78 24 1 14 SIGNIFICANCE 10 24 1 11 SIGNIFICANT 56 24 1 1 SIMMONS, MELVIN K. 109 70 1 2 SINGH, B.N. 54 70 1 1 SIOLI, HAROLD 39 70 1 1 SIOLI, HAROLD 75 70 1 1 SKILLED WORKERS 86 69 1 14 SLATYER, R.O. 10 70 1 1 SLATYER, R.O. 13 70 1 2 SLAVIK, B. 7 70 1 1 SOCIAL 47 24 1 4 SOCIAL 111 24 1 4 SOCIAL CHANGE 85 69 1 5 SOCIAL CHANGE 111 69 1 3 SOCIAL ENVIRONMENT 117 69 1 6 SOCIAL FACTORS 82 69 1 8 SOCIAL INTEGRATION 125 69 1 9 SOCIAL LIFE 103 69 1 7 SOCIAL PARTICIPATION 91 69 1 17 SOCIAL PLANNING 111 69 1 8 SOCIAL REFORM 92 69 1 4 SOCIAL SCIENCE RESEARCH 82 69 1 9 SOCIAL SECURITY 101 69 1 1 SOCIAL SECURITY 151 69 1 1 SOCIAL STATUS 95 69 1 2 SOCIAL STRATIFICATION 95 69 1 16 SOCIAL SYSTEMS 91 69 1 7 SOCIALES 101 24 1 5 SOCIALES 151 24 1 5 SOCIALISM 91 69 1 8 SOCIO-ECONOMIC FACTORS 86 69 1 12 SOCIO-ECONOMIC FACTORS 108 69 1 5 SOCIOLOGY 95 69 1 25 SOCIOLOGY 112 69 1 3 SOCIOLOGY OF COMMUNICATION 135 69 1 1 SOCIOLOGY OF SCIENCE 87 69 1 6 SOEKMONO, R. 94 70 1 1 SOIL 9 69 1 2 SOIL 20 24 1 7 SOIL 21 24 1 15 SOIL 21 24 1 20 SOIL 40 24 1 5 SOIL 54 24 1 4 SOIL 64 24 1 6 SOIL EROSION 64 69 1 2 SOIL EROSION 78 69 1 6 SOIL FERTILITY 38 69 1 2 SOIL FERTILITY 40 69 1 2 SOIL FERTILITY 54 69 1 1 SOIL GENESIS 41 69 1 1 SOIL MECHANICS 40 69 1 1 SOIL MOISTURE 5 69 1 2 SOIL MOISTURE 17 69 1 2 SOIL MOISTURE 18 69 1 2 SOIL MOISTURE 19 69 1 4 SOIL MOISTURE 20 69 1 4 SOIL MOISTURE 21 69 1 2 SOIL MOISTURE 30 69 1 4 SOIL SCIENCE 28 69 1 4 SOIL SURVEYS 31 69 1 4 SOIL SURVEYS 34 69 1 5 SOIL SURVEYS 38 69 1 1 SOIL SURVEYS 48 69 1 5 SOIL TYPES 20 69 1 2 SOIL TYPES 40 69 1 4 SOIL TYPES 41 69 1 6 SOILS 18 24 1 9 SOILS 29 69 1 7 SOILS 38 24 1 1 SOILS 39 24 1 1 SOILS 39 69 1 1 SOILS 41 24 1 8 SOILS 50 24 1 8 SOILS 50 69 1 3 SOLAR ENERGY 141 69 1 2 SOLS 29 24 1 12 SOME 63 24 1 1 SOME 70 24 1 1 SOUTERRAINES 33 24 1 8 SOUTH EAST ASIA 47 69 1 5 SOUTH EAST ASIA 133 69 1 4 SOUTHERN AFRICA 126 69 1 3 SPAIN 118 69 1 3 SPECIAL 71 24 1 3 SPORT 119 69 1 15 SRI LANKA 61 69 1 6 STAKE, ROBERT E. 136 70 1 1 STANDARD 131 24 1 9 STANDARDIZATION 105 69 1 22 STATEMENT 148 24 1 2 STATEMENTS 132 24 1 7 STATES 115 24 1 7 STATISTICAL ANALYSIS 120 69 1 1 STATISTICAL ANALYSIS 123 69 1 1 STATISTICAL ANALYSIS 124 69 1 1 STATISTICAL ANALYSIS 131 69 1 5 STATISTICAL DATA 86 69 1 11 STATISTICAL DATA 95 69 1 24 STATISTICAL DATA 100 69 1 12 STATISTICAL DATA 147 69 1 5 STATISTICAL DATA 150 69 1 12 STATISTICAL METHODOLOGY 105 69 1 4 STATISTICS 105 24 1 1 STATUS 10 24 1 19 STATUS 40 24 1 2 STATUS 95 24 1 3 STELLING-MICHAUD, SVEN 98 70 1 1 STOMATAL 9 24 1 8 STONE, E.C. 18 70 1 1 STORM 77 24 1 9 STORMS 77 69 1 8 STORMS 80 69 1 6 STRATEGIES 130 24 1 4 STRATEGY 110 24 1 3 STRESS 5 24 1 15 STRESS 11 24 1 4 STRUCTURAL 45 24 1 2 STUDENT ATTITUDE 86 69 1 19 STUDENT BEHAVIOUR 90 69 1 4 STUDENT SELECTION 88 69 1 6 STUDENT SOCIOLOGY 86 69 1 18 STUDIES 8 24 1 1 STUDIES 9 24 1 15 STUDIES 34 24 1 8 STUDIES 37 24 1 3 STUDIES 73 24 1 5 STUDIES 97 24 1 3 STUDIES 99 24 1 1 STUDY 5 24 1 9 STUDY 19 24 1 4 STUDY 78 24 1 2 STUDY 116 24 1 7 SUCTION 14 24 1 13 SUD 29 24 1 19 SULPHATE 41 24 1 7 SUNDERBANS 66 24 1 5 SUNDERBANS 67 24 1 6 SUNDERBANS 68 24 1 10 SUR 29 24 1 1 SUR 30 24 1 2 SUR 81 24 1 15 SUR 124 24 1 22 SURFACE 21 24 1 12 SURFACE 58 24 1 1 SURGES 77 24 1 10 SURVEY ANALYSIS 131 69 1 6 SURVEYS 105 69 1 5 SURVEYS 131 24 1 12 SURVIVAL 18 24 1 7 SWITZERLAND 98 69 1 6 SYMPOSIUM 28 24 1 16 SYMPOSIUM 112 24 1 7 SYSTEMS ANALYSIS 83 69 1 8 SYSTEMS ANALYSIS 87 69 1 11 TAIWAN 78 69 1 8 TEACHER BEHAVIOUR 93 69 1 5 TEACHER EDUCATORS 127 69 1 5 TEACHER GUIDES 93 69 1 8 TEACHER GUIDES 127 69 1 4 TEACHER ROLE 93 69 1 4 TEACHER ROLE 119 69 1 8 TEACHER TRAINING 114 69 1 8 TEACHER TRAINING 119 69 1 9 TEACHER TRAINING 127 69 1 3 TEACHING METHODS 93 69 1 2 TEACHING METHODS 127 69 1 7 TEACHING METHODS 149 69 1 8 TEACHING SKILLS 93 69 1 6 TECHNICAL 131 24 1 6 TECHNICAL AND VOCATIONAL EDUCA 149 69 1 10 TECHNICAL ASSISTANCE 132 69 1 3 TECHNICAL PERSONNEL 105 69 1 13 TECHNICAL TRAINING 134 69 1 2 TECHNIQUE 10 24 1 6 TECHNIQUE 81 24 1 1 TECHNIQUES 1 24 1 1 TECHNIQUES 93 24 1 3 TECHNOLOGIES 141 24 1 9 TECHNOLOGY 105 24 1 5 TECHNOLOGY ASSESSMENT 131 69 1 1 TECHNOLOGY TRANSFER 118 69 1 7 TECHNOLOGY TRANSFER 141 69 1 5 TECHNOLOGY TRANSFER 147 69 1 10 TELEVISION 91 69 1 14 TELEVISION 96 24 1 8 TELEVISION 113 69 1 11 TELEVISION PROGRAMMES 96 69 1 7 TEMPERATE 37 24 1 11 TEMPERATURE 3 24 1 14 TEMPERATURE 3 69 1 3 TEMPERATURE 81 24 1 21 TEMPERATURE 81 69 1 5 TEMPLES 94 69 1 2 TEMPS 129 24 1 18 TENDANCES 120 24 1 10 TENTATIVE 76 24 1 1 TEREDINID 68 24 1 3 TERMS 24 24 1 12 TERRITORY 45 24 1 10 TEXTURALES 29 24 1 4 THAILAND 37 24 1 26 THAILAND 37 69 1 5 THAILAND 58 69 1 9 THAILAND 78 69 1 10 THEBAUD, SCHILLER 105 70 1 1 THEIR 28 24 1 10 THEIR 62 24 1 10 THEIR 69 24 1 4 THEORY 100 24 1 3 THEORY 150 24 1 3 THERMAL PROPERTIES OF SUBSTANC 15 69 1 1 THREE 13 24 1 3 TIDAL 80 24 1 3 TIDES 43 24 1 11 TIDES 44 69 1 3 TIDES 57 69 1 3 TIDES 74 69 1 4 TIDES 77 24 1 4 TIDES 77 69 1 4 TIDES 80 69 1 2 TIME 117 24 1 4 TISON, LEON J. 32 70 1 1 TOOL 5 24 1 6 TOPICS 119 24 1 13 TOPOGRAPHY 75 69 1 5 TOPOGRAPHY 78 69 1 3 TOPOLOGY 46 69 1 4 TOWNS 95 69 1 7 TRACTS 40 24 1 6 TRADITIONAL 50 24 1 5 TRAINING 85 24 1 1 TRAINING 85 24 1 11 TRAINING 93 24 1 2 TRAINING 134 24 1 1 TRAINING 149 24 1 3 TRAINING ABROAD 125 69 1 2 TRAINING CENTRES 134 69 1 1 TRAINING COURSES 85 69 1 3 TRAINING COURSES 134 69 1 3 TRANSNATIONAL CORPORATIONS 110 69 1 1 TRANSNATIONAL CORPORATIONS 111 69 1 1 TRANSNATIONAL CORPORATIONS 147 69 1 11 TRANSPIRANTS 5 24 1 2 TRANSPIRATION 1 24 1 6 TRANSPIRATION 2 24 1 14 TRANSPORT 37 24 1 2 TRANSPORT 73 69 1 5 TRANSPORT 74 69 1 9 TRICART, J. 31 70 1 1 TROPICAL 28 24 1 6 TROPICAL 37 24 1 13 TROPICAL 52 24 1 13 TROPICAL 55 24 1 7 TROPICAL 65 24 1 6 TROPICAL 71 24 1 10 TROPICAL ZONES 31 69 1 3 TROPICAL ZONES 34 69 1 3 TROPICAL ZONES 41 69 1 4 TROPICAL ZONES 50 69 1 9 TROPICAL ZONES 52 69 1 2 TROPICAL ZONES 54 69 1 5 TROPICAL ZONES 55 69 1 4 TROPICAL ZONES 60 69 1 3 TROPICAL ZONES 63 69 1 5 TROPICAL ZONES 65 69 1 4 TROPICAL ZONES 71 69 1 5 TROPICAUX 31 24 1 16 TROPICS 41 24 1 16 TROPICS 54 24 1 9 TROPICS 64 24 1 12 TROPICS 69 24 1 2 TURGIDITY 10 24 1 5 TURKEY 107 69 1 4 TWEETEN, LUTHER G. 100 70 1 1 TWEETEN, LUTHER G. 150 70 1 1 TYPE 9 24 1 3 TYPOLOGY 110 24 1 5 TYPOLOGY 110 69 1 7 U 45 24 1 13 UNDER 6 24 1 11 UNDER 37 24 1 10 UNDERDEVELOPMENT 128 69 1 8 UNESCO 102 24 1 1 UNESCO 105 24 1 11 UNESCO 138 24 1 14 UNESCO 138 24 1 15 UNITED 79 24 1 5 UNITED 115 24 1 6 UNITED 116 24 1 2 UNIVERSALS 82 24 1 3 UNIVERSIDAD 128 24 1 3 UNIVERSITES 98 24 1 7 UNIVERSITIES 97 24 1 5 UNIVERSITIES 98 69 1 1 UNIVERSITIES 128 69 1 2 UNIVERSITIES 148 69 1 2 UNIVERSITY 88 24 1 2 UNIVERSITY COOPERATION 119 69 1 6 UNIVERSITY COURSES 88 69 1 1 UNIVERSITY CURRICULUM 88 69 1 3 UNIVERSITY EXTENSION 84 69 1 11 URBAN AREAS 84 69 1 7 URBAN AREAS 104 69 1 16 URBAN DEVELOPMENT 73 69 1 3 URBAN LIFE 103 69 1 6 URBAN PLANNING 46 69 1 1 URBAN POPULATION 72 69 1 4 URBAN RENEWAL 103 69 1 8 URUGUAY 105 69 1 10 USA 18 69 1 5 USA 32 69 1 8 USA 90 69 1 5 USA 100 69 1 3 USA 109 69 1 2 USA 114 69 1 5 USA 115 69 1 5 USA 139 69 1 2 USA 150 69 1 3 USE 9 24 1 12 USE 34 24 1 2 USE 73 24 1 4 USING 37 24 1 4 USSR 36 69 1 8 USSR 43 69 1 5 USSR 45 69 1 4 V 109 24 1 5 VALEUR 31 24 1 13 VAPOUR 4 24 1 8 VAPOUR 13 24 1 4 VARIABILITY 15 24 1 9 VASAK, KAREL 97 70 1 1 VEGETATION 27 69 1 4 VEGETATION 28 69 1 7 VEGETATION 30 24 1 4 VEGETATION 30 69 1 1 VEGETATION 39 69 1 7 VEGETATION 46 69 1 3 VEGETATION 50 24 1 10 VEGETATION 50 69 1 2 VEGETATION 55 24 1 1 VEGETATION 55 69 1 1 VEGETATION 60 24 1 6 VEGETATION 60 69 1 1 VEGETATION 61 24 1 3 VEGETATION 61 69 1 1 VEGETATION 62 69 1 6 VEGETATION 63 69 1 7 VEGETATION 69 24 1 5 VEGETATION 69 69 1 1 VEGETATION 78 69 1 7 VEGETAUX 81 24 1 14 VELOCITY 3 24 1 13 VERSLUYS, J.D.N. 47 70 1 1 VERSTAPPEN, H.TH. 34 70 1 1 VICE 88 24 1 9 VIET NAM SR 30 69 1 8 VIET NAM SR 41 69 1 7 VIET NAM SR 58 69 1 10 VILLAGE 90 24 1 12 VILLAGES 96 69 1 6 VIOLENCE 142 24 1 3 VISSER, W.C. 20 70 1 1 VITALITY 8 24 1 10 VOLKER, ADRIAAN 58 70 1 1 VOLKER, ADRIAAN 74 70 1 1 VOLKER, ADRIAAN 76 70 1 1 VULNERABLE 107 24 1 3 WANIEWICZ, IGNACY 117 70 1 1 WAR 140 69 1 3 WATER 4 24 1 7 WATER 5 24 1 14 WATER 10 24 1 18 WATER 11 24 1 3 WATER 12 24 1 6 WATER 13 24 1 8 WATER 13 69 1 2 WATER 14 24 1 5 WATER 16 69 1 3 WATER 22 24 1 1 WATER 24 24 1 15 WATER 25 24 1 3 WATER 43 24 1 6 WATER 52 24 1 8 WATER 57 24 1 2 WATER BALANCE 4 69 1 4 WATER BALANCE 11 69 1 2 WATER BALANCE 14 69 1 1 WATER BALANCE 22 69 1 1 WATER BALANCE 24 69 1 2 WATER BALANCE 25 69 1 2 WATER YIELD 22 69 1 2 WAVES 77 24 1 7 WAY 90 24 1 3 WEINRICH, A.K.H. 95 70 1 1 WENT, F.W. 4 70 1 2 WEST 35 24 1 9 WEST 68 24 1 11 WESTERN 37 24 1 16 WESTERN EUROPE 37 69 1 4 WETLANDS 55 69 1 8 WIDE 87 24 1 6 WILTING 18 24 1 12 WIND 3 24 1 12 WIND 3 69 1 4 WIND POWER 141 69 1 11 WOMEN 95 69 1 18 WOOD 68 24 1 1 WOODWELL, GEORGE M. 6 70 1 2 WOODWELL, GEORGE M. 27 70 1 1 WORKERS EDUCATION 84 69 1 9 WORKING POPULATION 149 69 1 12 WORKS 49 24 1 6 WORLD 87 24 1 5 WORLD 143 24 1 5 WORLD 147 24 1 8 WORLDWIDE 110 24 1 9 WYNTER, HECTOR 138 70 1 2 Y 118 24 1 2 Y 128 24 1 2 YIELDS 22 24 1 2 YOSUFZAI, H.K. 70 70 1 1 YOUTH 120 69 1 15 YOUTH ORGANIZATIONS 126 69 1 2 YUGOSLAVIA 91 24 1 4 YUGOSLAVIA 91 69 1 3 ZACKLIN, RALPH 116 70 1 1 ZAMBIA 86 24 1 8 ZAMBIA 86 69 1 2 ZIMBABWE 95 69 1 5 ZIMBABWE 116 69 1 2 ZONE 28 24 1 7 ZONE 71 24 1 11 test/cds.mrd0100644000076400001440000016054210033252173012144 0ustar gabiusers44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Techniques for the measurement of transpiration of individual plants 26 aParis bUnesco c-1965 30 ap. 211-224 billus. 70 Magalhaes, A.C. 70 Franco, C.M. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Controlled climate in the plant chamber and its influence upon assimilation and transpiration 26 c1965 30 ap. 225-232 billus. 70 Bosian, G. 24 Control of conditions in the plant chamber: fully automatic regulation of wind velocity, temperature and relative humidity to conform to microclimatic field conditions 26 c1965 30 ap. 233-238 billus. 70 Bosian, G. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Electric hygrometer apparatus for measuring water-vapour loss from plants in the field 26 c1965 30 ap. 247-257 billus. 70 Grieve, B.J. 70 Went, F.W. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 24 Anti-transpirants as a research tool for the study of the effects of water stress on plant behaviour 26 c1965 30 ap. 269-274 billus. 70 Gale, J. 70 Poljakoff-Mayber, A. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Measurements of plant carbon dioxide exchange by infra-red absorption under controlled conditions and in the field 26 c1965 30 ap. 283-289 70 Bourdeau, Philippe F. 70 Woodwell, George M. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Colorimetric determination of CO2 exchange in field and laboratory 26 c1965 30 ap. 291-298 billus. 70 Slavk, B. 70 Catsky, J. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Studies on gaseous exchange physiology with regard to plant vitality 26 c1965 30 ap. 299-306 billus. 70 Polster, H. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 New type of porometer for measuring stomatal opening and its use in irrigation studies 26 c1965 30 ap. 325-329 billus. 24 Modifications to the relative turgidity technique with notes on its significance as an index of the internal water status of leaves 25 20 25 30 25 40 26 c1965 30 ap. 331-342 billus. 70 Slatyer, R.O. 70 Barrs, H.D. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Measurement of water stress in plants 26 c1965 30 ap. 343-351 billus. 70 Kramer, P.J. 70 Brix, H. 24 Leaf-disc method for determining water saturation deficit 26 c1965 30 ap. 353-360 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 70 Catsky, J. 24 Experience with three vapour methods for measuring water potential in plants 26 c1965 30 ap. 369-384 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Barrs, H.D. 70 Slatyer, R.O. 24 Determination of the internal water balance (hydrature) in the field by measuring suction force and refractive index 26 c1965 30 ap. 385-391 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Kreeb, K. 24 Heat resistance of plants, its determination and variability 26 c1965 30 ap. 399-405 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 70 Lange, O.L. 24 Measurement of drought resistance 26 c1965 30 ap. 407-412 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 70 Levitt, J. 24 Measurement of drought resistance in crop plants 26 c1965 30 ap. 413-419 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Role of dew in pine survival in soils below the wilting point and its measurement 26 c1965 30 ap. 421-427 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Stone, E.C. 24 Gauges for the study of evapotranspiration rates 26 c1965 30 ap. 443-451 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Hudson, J.P. 24 Method of determining evapotranspiration in soil monoliths 26 c1965 30 ap. 453-460 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Visser, W.C. 24 Determination of the evaporation from the plant cover and the surface of the soil by relating lysimeter and soil moisture measurements to potential evaporation 26 c1965 30 ap. 461-465 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Klausing, O. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 24 Water yields of forest, Maquis and grass covers in semi-arid regions: a literature review 26 c1965 30 ap. 467-477 billus. 70 Shachori, A.Y. 70 Michaeli, A. W 24 24 Calculation model for the actual evapotranspiration from cropped areas and other terms of the water balance equation 26 c1965 30 ap. 489-494 billus. 44 Methodology of plant eco-physiology: proceedings 69 70 Makkink, G.F. 70 Heemst, H.D.J. van 24 Heat, water and carbon dioxide budget of plant cover: methods and measurements 26 c1965 30 ap. 495-512 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Baumgartner, Albert 24 Indirect methods of measurement of dry matter production 26 c1965 30 ap. 513-518 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Lieth, H. 24 Measurement of dry matter production of the plant cover 26 c1965 30 ap. 519-527 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 Paper on: 70 Woodwell, George M. 70 Bourdeau, Philippe F. 44 Humid tropics research 50 Incl. bibl. 69 Proceedings on: 71 Pakistan Academy of Sciences 71 Council of Scientific and Industrial Research (Pakistan) 71 Association for the Advancement of Science (Pakistan) 71 University of Dacca 72 Symposium on Scientific Problems of the Humid Tropical Zone Deltas and their Implications pDacca d1964 76 Les Problmes scientifiques des deltas de la zone tropicale humide et leurs implications: actes du Colloque de Dacca zfre 24 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 26 aParis bUnesco c1966 30 a422 p. billus., maps c27 cm. 24 Sur les caractristiques texturales et minralogiques des sdiments rcents et des sols alluviaux dans le delta du Mangoky, sud-ouest de Madagascar 26 c1966 30 ap. 73-82 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Hervieu, J. 24 Apercu sur la vgtation occupant les alluvions rcentes 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Schmid, Maurice 24 Place de la gomorphologie dans l'tude de la mise en valeur des deltas tropicaux 26 c1966 30 ap. 15-22 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Tricart, J. 24 Problmes de sdimentation dans les deltas 26 c1966 30 ap. 41-57 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Tison, Lon J. 24 Concentration chimique par vaporation des eaux souterraines dans les deltas 26 c1966 30 ap. 169-173 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Schoeller, Marc 24 Use of aerial photographs in delta studies 26 c1966 30 ap. 29-33 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Verstappen, H.Th. 24 On the gradual shifting of the Ganges from west to east in delta-building operations 26 c1966 30 ap. 35-40 bmaps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Chowdhury, M.I. 24 Hydrology and formation of river-mouth bars 26 c1966 30 ap. 59-64 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Mikhailov, V.N. 24 Mud transport studies using manganese as an accompanying element under temperate and tropical climatic conditions: Western Europe, Amazon area and Chao-Phya river area in Thailand 26 c1966 30 ap. 65-71 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Groot, A.J. de 24 Soils of East Pakistan 26 c1966 30 ap. 83-87 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Islam, M. Amirul 24 Soils in the estuary of the Amazon 26 c1966 30 ap. 89-96 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Sioli, Harold 24 Nutrient status of different soil tracts of East Pakistan 26 c1966 30 ap. 97-101 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Karim, A.Q.M.B. 24 Morphology, genesis and occurence of acid sulphate soils, cat-clays in deltas of the humid tropics 26 c1966 30 ap. 103-106 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Moormann, F.R. 24 Meteorological problems of the deltaic flood-plains of East Pakistan 26 c1966 30 ap. 123-133 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Naqvi, Sibte Nabi 24 Interaction of river and sea water in the absence of tides 26 c1966 30 ap. 175-180 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Borichansky, L.S. 70 Mikhailov, V.N. 24 Hydraulic scale models of deltas, including mathematical models 26 c1966 30 ap. 181-188 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Nagaraja, V.N. 24 Main structural features of the great deltas in the territory of the U.S.S.R. 26 c1966 30 ap. 393-397 70 Egorov, V.V. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 24 Methodology in area analysis 26 c1966 30 ap. 361-369 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Crain, N. 24 Hydrological achievements and social problems 26 c1966 30 ap. 357-359 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Versluys, J.D.N. 24 Progress of the river deltas in Java 26 c1966 30 ap. 347-355 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Hollerw<=oe>ger, F. 24 Influence of hydraulic improvement works 26 c1966 30 ap. 341-345 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 70 Deguchi, K. 24 Influence of man's traditional activities on soils and vegetation 26 c1966 30 ap. 301-304 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Boesch, H. 24 Deltas: a basis of culture and civilization 26 c1966 30 ap. 295-300 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 B<=ue>del, J. 24 Biology of fresh, brackish and saline water as it contributes to tropical delta formation 26 c1966 30 ap. 253-256 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Doty, M.S. 70 Mueller-Dombois, D. 24 Biology of estuaries 26 c1966 30 ap. 257-261 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Caspers, H. 24 Problems presented by soil microbiology in the humid tropics with reference to the deltas 26 c1966 30 ap. 249-251 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Singh, B.N. 24 Vegetation as a geological agent in tropical deltas 26 c1966 30 ap. 227-233 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Fosberg, F. Raymond 24 Significant changes of river regime due to delta growth 26 c1966 30 ap. 151-156 billus., map 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 24 Ground water in deltas 26 c1966 30 ap. 157-168 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Huisman, L. 24 Surface hydrology of deltaic areas 26 c1966 30 ap. 143-150 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Volker, Adriaan 24 Control of floods in East Pakistan 26 c1966 30 ap. 135-141 bmaps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Abbas, B.M. 24 General features of East Pakistan vegetation 26 c1966 30 ap. 201-205 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Ahmad, Majeed 24 Estuarine vegetation of Ceylon 26 c1966 30 ap. 207-210 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Abeywickrama, B.A. 24 Estuaries of the Amazon and Oiapoque rivers and their floras 26 c1966 30 ap. 211-218 billus., map 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Pires, J. Mura 24 Some factors involved in mangrove establishment 26 c1966 30 ap. 219-225 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Chapman, Valentine J. 24 Influence of forests on the soil and microclimate in the humid tropics 26 c1966 30 ap. 243-246 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Ghani, Q. 24 Plankton productivity in relation to tropical fish culture 26 c1966 30 ap. 263-266 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 paper on: 70 Prowse, G.A. 24 Fish fauna of the Sunderbans 26 c1966 30 ap. 267-269 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Qureshi, M. Rahimullah 24 Fish and fisheries of the Sunderbans 26 c1966 30 ap. 271-276 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Ahmad, Nazir 24 Wood-boring Teredinid shipworms Mollusca of mangroves in the Sunderbans, West Bengal 26 c1966 30 ap.277-283 billus., map 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Roonwal, M.L. 24 Humid tropics and their vegetation 26 c1966 30 ap. 191-199 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Khuda, M. Qudrat 24 Some important animal pests and parasites of East Pakistan 26 c1966 30 ap. 285-291 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Yosufzai, H.K. 24 Agriculture with special reference to rice cultivation in humid tropical zone deltas 26 c1966 30 ap. 305-316 44 Scientific problems of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Goor, G.A.W. van de 24 Population of East Pakistan 26 c1966 30 ap. 321-328 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Patel, A.M. 24 Need for land-use studies in East Pakistan 26 c1966 30 ap. 329-332 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Ahmad, Nazir 70 Khan, F. Karim 24 Deltaic area of the Irrawaddy river in Burma 26 c1966 30 ap. 373-379 billus., map 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Volker, Adriaan 24 General features of the delta of the Amazon 26 c1966 30 ap. 381-390 billus., maps 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Sioli, Harold 24 Tentative classification and comparison with deltas of other climatic regions 26 c1966 30 ap. 399-408 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 50 Incl. bibl. 69 Paper on: 70 Volker, Adriaan 24 Effect of tides, coastal currents, waves and storm surges on the natural conditions prevailing in deltas 26 c1966 30 ap. 109-113 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Santema, P. 24 Comparative study on the geomorphology and flooding in the plains of the Cho-Shui-Chi, Chao-Phya, Irrawaddy and Ganges 26 c1966 69 Paper on: 70 Ohya, Masahiko 76 La Politique culturelle en Rpublique unie du Cameroun zfre 44 Studies and documents on cultural policies 50 Incl. bibl. 69 - 24 Cultural policy in the United Republic of Cameroon 26 aParis bUnesco Press c1976 30 a91 p. billus. c24 cm. 70 Bahoken, J.C. 70 Atangana, Engelbert 24 Hydrometeorological and tidal problems of the deltaic areas in India 26 c1966 30 ap. 115-121 billus. 44 Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium 69 Paper on: 70 Nagaraja, V.N. 24 Technique pour la mesure de l'mission de gaz carbonique par les organes vgtaux sur pied, en conditions dfinies de temprature et d'humidit relative 26 c1965 30 ap. 275-281 billus. 44 Methodology of plant eco-physiology: proceedings of the Montpellier Symposium 50 Incl. bibl. 69 70 Leblond, C. 70 Carlier, G. 24 Cross-cultural universals of affective meaning 26 aUrbana bUniversity of Illinois c1975 30 a486 p. billus. 50 Incl. bibl. 69 - 70 Osgood, Charles Egerton 70 May, William H. 70 Miron, Murray S. 24 Politics of organizational decision-making 26 aLondon bTavistock c1973 30 a302 p. billus. 50 Incl. bibl. 69 - 70 Pettigrew, Andrew M. 24 Adult education in India; a book of readings 26 aBombay bNachiketa Publications c1973 30 a532 p. 50 Incl. bibl. 69 - 70 Bordia, Anil 70 Kidd, James Robbins 70 Draper, James A. 71 Indian Adult Education Association 24 Training for change agents; a guide to the design of training programs in education and other fields 26 aAnn Arbor bCenter for Research on Utilization of Scientific Knowledge, University of Michigan c1973 30 a249 p. billus. 69 - 70 Havelock, Ronald G. 70 Havelock, Mary C. 71 University of Michigan (USA). Center for Research on Utilization of Scientific Knowledge 72 Conference on Educational Change Agent Training pClinton, USA d1970 24 Higher education and the labour market in Zambia: expectations and performance 26 aParis bUnesco Press bIIEP aLusaka bUniversity of Zambia 30 a373 p. billus., map c24 cm. 50 Incl. bibl. 69 - 70 Sanyal, Bikas C. 70 Case, John H. 70 Dow, Philip S. 70 Jackman, Mary Elizabeth 71 University of Zambia 71 IIEP 24 Innovation in science education - world-wide 26 aParis bUnesco Press c1976 30 a249 p. billus. c24 cm. 70 Baez, Albert V. 50 Incl. bibl. 69 - 76 L'Innovation dans l'enseignement des sciences: synthse mondiale zfre 24 Open university; a personal account by the first Vice-Chancellor 26 aMilton Keynes, Eng. bOpen University Press c1976 30 a298 p. 69 70 Perry, Walter 74 History and evaluation of a dynamic innovation in higher education 24 Classical Greece 26 aNew York bTime-Life Books c1965 30 a192 p. billus., maps 50 Incl. bibl. 69 - 70 Bowra, Cecil Maurice 24 Long way from home; effects of public high schools on village children away from home 26 aFairbanks, Alaska bCenter for Northern Educational Research, University of Alaska c1973 30 a1 v. (various pagings) 50 (ISEGR report no.38). Incl. bibl. 69 - 70 Kleinfeld, Judith S. 70 Bloom, Joseph 71 University of Alaska (USA). Center for Northern Educational Research 71 University of Alaska (USA). Institute of Social, Economic and Government Research 74 Effects of public high schools on village children away from home 24 Communication policies in Yugoslavia 26 aParis bUnesco c1976 30 a66 p. c24 cm. 50 Incl. bibl. 69 - 70 Lekovic, Zdravko 70 Bjelica, Mihalo 71 Yugoslav Institute of Journalism 76 Les Politiques de la communication en Yougoslavie zfre 24 Peruvian model of innovation: the reform of basic education 26 aGeneva bIBE c1976 30 a53 p. billus. c22 cm. 70 Churchill, Stacy 70 Guttmann, Ludwig 44 Experiments and innovations in education v22 50 Incl. bibl. 69 - 71 IBE 76 El Modelo peruano de innovacin: la reforma de la educacin bsica zspa 24 Broadcast training techniques 26 aParis bUnesco Press c1976 30 a122 p. billus. c22 cm. 50 Incl. bibl. 69 - 70 Dance, F. Robert 76 Tcnicas de formacin del personal de comunicacin zspa 24 Chandi Borobudur; a monument of mankind 26 aParis bUnesco Press aAmsterdam bVan Gorcum c1976 30 a53 p. billus., map, plans c24 cm. 50 Incl. bibl. 69 - 70 Soekmono, R. 76 Chandi Borobudur; un monument pour toute l'humanit zfre 24 Mucheke: race, status and politics in a Rhodesian community 26 aParis bUnesco c1976 30 a278 p. billus., map, plans c24 cm. 50 Companion volume to both Racism and apartheid in southern Africa: Rhodesia, 1975, and to Southern Rhodesia; the effects of a conquest society on education, culture and information (in preparation). Incl. bibl. 69 - 70 Weinrich, A.K.H. 24 Planning for satellite broadcasting: the Indian Instructional Television Experiment 26 aParis bUnesco Press c1976 30 a71 p. billus., map c27 cm. 44 Reports and papers on mass communication v78 50 Incl. bibl. 69 - 76 Planification de l'utilisation des satellites des fins ducatives: L'Exprience indienne de tlvision ducative par satellite zfre 76 Planeamiento de radiodifusin por satlite: El Experimento de Televisin Educativa en la India zspa 24 Human rights studies in universities 26 aParis bA. Pdone c1973 30 a159 p. 50 (Human rights journal, v.6-1, 1973). "English summary of Enseignement des droits de l'homme dans les universits..." Incl. bibl. 69 Report on: 70 Vasak, Karel 71 International Institute of Human Rights (France) 71 International Law Association 74 Human rights journal 74 Unesco-sponsored programmes and publications 76 Enseignement des droits de l'homme dans les universits zfre 24 Bibliographie internationale de l'histoire des universits - 2: Portugal - Leiden - Pcs - Franeker - Basel 26 aGenve bDroz c1976 30 ax, 219 p. 50 (Commission internationale pour l'histoire des universits. Etudes et travaux, 5) 69 Bibliography on: 70 Stelling<-= >Michaud, Sven 70 S, Artur Moreira de 71 International Commission for History of Universities 71 International Council for Philosophy and Humanistic Studies 74 Unesco-sponsored programmes and publications 24 Studies in the philosophy of biology: reduction and related problems 26 aLondon bMacMillan c1974 30 a390 p. 50 Papers presented at a conference on problems of reduction in biology, held in Villa Serbelloni, Italy, 1972. Incl. bibl. 69 - 70 Ayala, Francisco Jose 70 Dobzhansky, Theodosius Grigorievich 24 Micropolitan development: theory and practice of greater-rural economic development 26 aAmes bIowa State University Press c1976 30 a456 p. bmap 50 Incl. bibl. 69 - 70 Tweeten, Luther G. 70 Brinkman, George Lorris 24 Manuel pratique des assurances sociales 26 aParis bDOCIS c1968 30 a1 v. (loose-leaf) 50 Kept up-to-date 69 71 Documentation des institutions 24 Unesco, an ideal in action 26 aParis bUnesco c1976 30 a15 p. billus. c23 cm. 69 76 L'Unesco, un idal en action zfre 76 La Unesco, un ideal en accin zspa 24 About the quality of life 26 aParis bUnesco c1976 30 a66 p. billus. c20x21 cm. 44 Unesco and its programme 69 - 70 Fradier, Georges 76 A propos de la qualit de la vie zfre 24 Mise en place de politiques ducatives: rle et mthodologie de la carte scolaire 26 aParis bPresses de l'Unesco bNathan aBruxelles bEditions Labor c1976 30 a325 p. billus., maps, plan c22 cm. 70 Hallak, Jacques 50 (Education 2000). Incl. bibl. 69 - 71 IIEP 76 El Mapa escolar: un instrumento de la poltica de educacin zspa 76 Les Statistiques de la science et de la technologie en Amrique latine zfre 76 Las Estadsticas de la ciencia y la tecnologa en Amrica Latina zspa 24 Statistics on science and technology in Latin America; experience with Unesco pilot projects, 1972-1974 26 aParis bUnesco Press c1976 30 a76 p. c27 cm. 70 Thbaud, Schiller 44 Statistical reports and studies v20 50 Incl. bibl. 69 - 72 Coloquio sobre los Aspectos Nacionales e Internacionales de los Sistemas de Datos para las Estadsticas relativas a la Ciencia y la Tecnologa en Amrica Latina pCaracas d1974 24 List of periodicals; current holdings in the (IBE) Documentation Centre as of September 1976 26 aParis bUnesco aGeneva bInternational Bureau of Education, Documentation Centre c1976 30 a175 p. c30 cm. 69 71 IBE. Documentation Centre 76 Liste de priodiques; collections du Centre de documentation (du BIE) septembre 1976 zfre 24 Cyprus: the vulnerable republic 25 2nd rev. ed. 26 aThessaloniki bInstitute for Balkan Studies c1975 30 a223 p. bports 69 70 Bitsios, Dimitri S. 71 Institute for Balkan Studies (Cyprus) 24 Comunicaao/incomunicaao no Brasil 26 aSao Paulo bEdioes Loyola c1976 30 a207 p. billus. 50 (Biblioteca Universitria de Comunicaao, vol.1). Incl. bibl. 69 - 70 Melo, Jos Marques de 71 Uniao Crista Brasileira de Comunicaao Social 24 Annual review of energy - v.1, 1976 26 aPalo Alto, Calif. bAnnual Review Inc. c1976 30 a793 p. billus. 50 Incl. bibl. 69 - 70 Hollander, Jack M. 70 Simmons, Melvin K. 24 Multinational product strategy; a typology for analysis of worldwide product innovation and diffusion 26 aNew York bPraeger Publishers c1976 30 a220 p. billus. 50 Incl. bibl. 69 - 70 Leroy, Georges 24 Multinational corporation and social change 26 aNew York bPraeger Publishers c1976 30 a234 p. 50 Incl. bibl. 69 - 70 Apter, David Ernest 70 Goodman, Louis Wolf 12 New York University. Institute of Philosophy n2nd Annual meeting pNew York d1958 24 Psychoanalysis, scientific method, and philosophy; a symposium 26 aNew York bNew York University Press c1971 30 a370 p. 50 Incl. bibl. 69 - 70 Hook, Sidney 24 Mass media in C.M.E.A. countries 26 aPrague bIOJ c1976 30 a255 p. 70 Horlamus, Sepp 50 Incl. bibl. 69 - 71 International Organization of Journalists 71 Council for Mutual Economic Assistance 24 Educational development; a new discipline for self-renewal 26 aEugene, Or. bCenter for the Advanced Study of Educational Administration c1973 30 a343 p. billus. 50 (CASEA monograph, no.24). Incl. bibl. 69 - 70 Hemphill, John Knox 70 Rosenau, Fred S. 71 Far West Laboratory for Educational Research and Development (USA) 71 University of Oregon (USA). Center for the Advanced Study of Educational Administration 24 Folklore from Africa to the United States; an annotated bibliography 26 aWashington bLibrary of Congress c1976 30 a161 p. billus. 69 - 70 Coughlan, Margaret N. 71 Library of Congress (USA) 24 United Nations and Rhodesia: a study in international law 26 aNew York bPraeger Publishers c1974 30 a188 p. 50 Incl. bibl. 69 - 70 Zacklin, Ralph 24 Demand for part-time learning in Ontario 26 aOntario bOECA c1976 30 a216 p. 50 Incl. bibl. 69 - 70 Waniewicz, Ignacy 71 Ontario Educational Communications Authority (Canada) 71 Ontario Institute for Studies in Education (Canada) 12 Seminario Iberoamericano de Poltica Cientfica y Tecnolgica pAlcal de Henares and Madrid d1974 24 (Ponencias y otras documentos) 26 aMadrid bServicio de Publicaciones del Ministerio de Educacon y Ciencia c1976 30 a244 p. 69 - 71 OAS 12 Meeting on the Implementation of the Recommendations of the Third Regional Conference of Ministers of Education and Ministers Responsible for Economic Planning in the Arab States pCairo d1976 24 Selected resolutions and recommendations of recent conferences and meetings of relevance to topics likely to be discussed at the meeting 26 c17 Sept. 1976 30 a25 p. 69 12 Meeting on the Implementation of the Recommendations of the Third Regional Conference of Ministers of Education and Ministers Responsible for Economic Planning in the Arab States pCairo d1976 24 Croissance des effectifs scolaires dans les pays arabes; tendances et projections jusqu'en 1985 26 cJuly 1976 30 a96 p. billus. 69 - 12 Meeting of Experts on the Content of Education in the Context of Life-long Education pParis d1975 24 Final report 26 c6 Feb. 1976 30 a(18 p. in various pagings) 69 - 12 Meeting of Experts on the Content of Education in the Context of Life-long Education pParis d1975 24 Analytical description of the features of life-long education as presented by Mr. Dave 26 c23 Oct. 1975 30 a5 p. 69 70 Fajardo, Jos Carlos 70 Dave, Ravindrakumar H. 12 Meeting on the Implementation of the Recommendations of the Third Regional Conference of Ministers of Education and Ministers Responsible for Economic Planning in the Arab States pCairo d1976 24 Aspects quantitatifs de la scolarisation du second degr dans les pays arabes 26 c27 Aug. 1976 30 a25 p. 69 - 70 Sammak, A. 71 Unesco Regional Office for Education in the Arab States (Lebanon) 12 Meeting on the Implementation of the Recommendations of the Third Regional Conference of Ministers of Education and Ministers Responsible for Economic Planning in the Arab States pCairo d1976 24 Dveloppement quantitatif de l'enseignement primaire dans les pays arabes la lumire de la rsolution de la Confrence de Marrakech sur l'enseignement obligatoire 26 c27 Aug. 1976 30 a(44 p. in various pagings) billus. 69 - 71 Unesco Regional Office for Education in the Arab States (Lebanon) 12 Regional Seminar on Training n2nd pDakar d1975 24 Final report 26 c25 June 1976 30 a(17 p. in various pagings) 69 - 71 Unesco Regional Office for Education in Africa (Senegal) 12 Meeting between Members of National Liberation Movements in Southern Africa and Leaders of Youth Organizations pAlgiers d1975 24 Final report 26 c11 Dec. 1975 30 a(11 p. in various pagings) 69 71 Algeria. Ministre de la jeunesse et des sports 69 - 71 IBE 12 Meeting of National Directors and Chief Technical Advisers of the Projects for the Training of Educational Personnel n7th pParis d1975 24 Report 26 c15 June 1976 30 a(88 p. in various pagings) billus. 69 - 24 Desarrollo y universidad 26 aLa Paz bLibrera Editorial "Juventud" c1975 30 a206 p. 70 Anaya, Ricardo 50 Incl. bibl. 69 24 Philosophy and the novel; philosophical aspects of Middlemarch, Anna Karenina, The Brothers Karamazov, A la recherche du temps perdu and the methods of criticism 26 aOxford, Eng. bClarendon Press c1975 30 a216 p. 12 Joint FAO/WHO Expert Committee on Nutrition n9th pRome d1974 24 Food and nutrition strategies in national development 26 aRome bFAO c1976 30 a64 p. 50 (FAO nutrition meetings report series, no.56). WHO technical report series, no.584 69 - 71 FAO 71 WHO 76 Stratgies de l'alimentation et de la nutrition dans le dveloppement national zfre 24 Measurement of scientific and technical activities - Proposed standard practice for surveys of research and experimental development 26 aParis bOECD c1976 30 a138 p. 69 71 OECD 74 Proposed standard practice for surveys of research and experimental development 76 La Mesure des activits scientifiques et techniques zfre 12 UNDP/ESCAP Regional Meeting on Technical Cooperation among Developing Countries pBangkok d1976 24 Conclusions and recommendations and extracts from statements 26 aBangkok bESCAP c1976 30 a76 p. 69 71 ESCAP 24 Consultancy services available in developing ESCAP countries 26 aBangkok bESCAP c1983 30 a590 p. 69 71 ESCAP 24 Training courses available in developing ESCAP countries 26 aBangkok bESCAP c1976 30 a222 p. 69 - 71 ESCAP 24 Communication 26 aLondon bLongman c1975 30 a229 p. 50 Incl. bibl. 69 - 70 Quail, Denis 24 Evaluating educational programmes; the need and the response 26 aParis bCERI, OECD c1976 30 a89 p. 50 Incl. bibl. 69 70 Stake, Robert E. 71 OECD. Centre for Educational Research and Innovation 76 L'Evaluation des programmes d'enseignement; ncessit et ractions zfre 24 Audit informatique 26 aParis bMasson c1976 30 a245 p. billus. 69 - 70 Saint Antonin, Claude 24 Addresses delivered at a memorial ceremony for Ren Maheu formerly Director-General of Unesco (1961-1974), Unesco House, 26 April 1976 26 aParis bUnesco c1976 30 a35 p. bport. c21 cm. 70 Jbor, Magda 70 Wynter, Hector 70 Faure, Edgar 71 Unesco. Director-General, 1974- (M'Bow, A.M.) 74 Ren Maheu in memoriam 76 Allocutions prononces lors d'une crmonie la mmoire de Ren Maheu zfre 24 Michigan's environmental future; a master plan for environmental education 26 aLansing, Mich. bGovernor's Environmental Education Task Force c1973 30 a90 p. billus. 69 - 71 Michigan (USA). Governor's Environmental Education Task Force 24 Armaments and disarmament in the nuclear age; a handbook 26 aStockholm bSIPRI c1976 30 a308 p. 50 Incl. bibl. 69 - 71 Stockholm International Peace Research Institute 24 Energy for rural development: renewable resources and alternative technologies for developing countries 26 aWashington bNational Academy of Sciences c1976 30 a306 p. billus. 50 (PB-260-606). Incl. bibl. 69 - 71 National Academy of Sciences (USA). Ad Hoc Panel on Renewable Energy Resources 71 National Research Council (USA) 71 National Academy of Sciences (USA). Board on Science and Technology for International Development 24 Conquest of violence: the Gandhian philosophy of conflict 25 Rev. ed. 26 aBerkeley bUniversity of California Press c1971 30 a271 p. 50 Incl. bibl. 69 70 Bondurant, Joan V. 24 Mathematics in a changing world 26 aLondon bHeinemann Educational Books c1974 30 a293 p. billus., maps 50 Incl. bibl. 69 - 70 Holt, Michael 70 Marjoram, D.T.E. 24 Interaction between science and philosophy 26 aAtlantic Highlands, N.J. bHumanities Press c1974 30 a481 p. 50 Papers from a Symposium held in Jerusalem in January 1971 in the honor of Professor Samuel Sambursky. Incl. bibl. 69 70 Elkana, Yehuda 71 Israel Academy of Sciences and Humanities 71 Hebrew University (Israel) 24 Phenomenology and natural existence 26 aAlbany bState University of New York Press c1973 30 a408 p. bport 50 "Essays in honor of Marvin Farber." Marvin Farber bibliography to 1970:47-53. Incl. bibl. 69 - 70 Riepe, Dale Maurice 24 Australian perspectives on lifelong education 26 aHawthorn bACER c1976 30 a104 p. 50 (Australian education review, no.6). Incl. bibl. 69 - 70 Duke, Christopher 71 Australian Council for Educational Research 24 Employment, growth and basic needs; a one-world problem 26 aGeneva bILO c1976 30 a177 p. 50 "Report of the Director-General." Incl. bibl. 69 - 71 ILO 72 Tripartite World Conference on Employment, Income Distribution and Social Progress and the International Division of Labour pGeneva d1976 76 L'Emploi, la croissance et les besoins essentiels; problme mondial zfre 76 Empleo, crecimiento y necesidades esenciales; problema mundial zspa 12 Symposium on the Psychological Bases of Programmed Learning pTbilisi, USSR d1976 24 Personal statement 26 c1976 30 a1 p. 69 70 Okatcha, F.M.M.O. 12 Symposium on the Psychological Bases of Programmed Learning pTbilisi, USSR d1976 24 For a training project based on programmed learning in the developing countries: objectives, means and resources, organization and management, development 26 c14 June 1976 30 a18 p. 69 - 70 Pocztar, Jerry 24 Micropolitan development: theory and practice of greater-rural economic development 26 aAmes bIowa State University Press c1976 30 a456 p. bmap 50 Incl. bibl. 69 - 70 Tweeten, Luther G. 70 Brinkman, George Lorris 24 Manuel pratique des assurances sociales 26 aParis bDOCIS c1968 30 a1 v. (loose-leaf) 50 Kept up-to-date 69 71 Documentation des institutions test/unicode.m0d0100644000076400001440000000130710142214132012702 0ustar gabiusers# $Id: unicode.m0d,v 1.2 2004/11/03 17:41:46 kripke Exp $ 4 C unicode-demo 4 N . , - ( ) 4 W 0 1 2 3 4 5 6 7 8 9 4 W a b c č d e f g h i j k l ł ll m n ñ o p q r s t u v w x y z ź 4 A A B C Č D E F G H I J K L Ł LL M N Ñ O P Q R S T U V W X Y Z Ź 4 M o ó 4 M ae ä Ä 4 M oe ö Ö 4 M ue ü Ü 4 M ss ß 4 W ʃ ʌ ʨ 4 M ' 4 W α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ ς σ τ υ φ χ ψ ω 4 A Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω 4 W а б в г д е ж з и й к л м н о п р с т у ф х ц ч ш щ ъ ы ь э ю я 4 A А Б В Г Д Е Ж З И Й К Л М Н О П Р С Т У Ф Х Ц Ч Ш Щ Ъ Ы Ь Э Ю Я 4 M α ά 4 M η ή test/unicode.mqt0100644000076400001440000000153610142162023013030 0ustar gabiusersAare 3 23 0 0 Berlin 8 21 0 0 Bern 3 21 0 0 Deutschland 5 22 0 0 Deutschland 6 22 0 0 Deutschland 7 22 0 0 Deutschland 8 22 0 0 Donau 1 23 0 1 Wien 1 23 0 2 España 14 22 0 0 Frankfurt 5 21 0 0 Frankfurt 6 21 0 0 Guadalquivir 14 23 0 0 Ilysos 12 23 0 1 Kephisos 12 23 0 2 Inn 2 23 0 0 Innsbruck 2 21 0 0 Köln 7 21 0 0 Limmat 4 23 0 0 Main 5 23 0 0 Moldau 11 23 0 1 Vltava 11 23 0 2 Moskwa 13 23 0 0 Oder 6 23 0 0 Polen 9 22 0 0 Polen 10 22 0 0 Praha 11 21 0 0 Rhein 7 23 0 0 Schweiz 3 22 0 0 Schweiz 4 22 0 0 Sevilla 14 21 0 0 Spree 8 23 0 1 Havel 8 23 0 2 Warszawa 10 21 0 1 var'ʃava 10 21 0 2 Wien 1 21 0 0 Wisła 10 23 0 0 Zürich 4 21 0 0 Österreich 1 22 0 0 Österreich 2 22 0 0 Česko 11 22 0 0 Łódź 9 21 0 1 wuʨ 9 21 0 2 Łódź 9 23 0 0 Αθήνα 12 21 0 0 Ελλάδα 12 22 0 0 Москва 13 21 0 1 mʌ'skva 13 21 0 2 Россия 13 22 0 0 test/unicode.mrd0100644000076400001440000000105010142162023013000 0ustar gabiusers21 Wien 22 Österreich 23 Donau, Wien 21 Innsbruck 22 Österreich 23 Inn 21 Bern 22 Schweiz 23 Aare 21 Zürich 22 Schweiz 23 Limmat 21 Frankfurt 22 Deutschland 23 Main 21 Frankfurt 22 Deutschland 23 Oder 21 Köln 22 Deutschland 23 Rhein 21 Berlin 22 Deutschland 23 Spree, Havel 21 Łódź [wuʨ] 22 Polen 23 Łódź 21 Warszawa [var'ʃava] 22 Polen 23 Wisła 21 Praha 22 Česko 23 Moldau (Vltava) 21 Αθήνα 22 Ελλάδα 23 Ilysos, Kephisos 21 Москва [mʌ'skva] 22 Россия 23 Moskwa 21 Sevilla 22 España 23 Guadalquivir