diff options
| author | Hyunjee Kim <hj0426.kim@samsung.com> | 2019-11-25 17:01:31 +0900 |
|---|---|---|
| committer | Hyunjee Kim <hj0426.kim@samsung.com> | 2019-11-25 17:01:31 +0900 |
| commit | 26ffbb3c93868dfc85b3933020b7e960bf0bd3a4 (patch) | |
| tree | c1630ab636ae3f282dcf019fd0c282044536b15e | |
| parent | 48da3963020bb71b5952d410358e9edc324c4562 (diff) | |
| download | python-26ffbb3c93868dfc85b3933020b7e960bf0bd3a4.tar.gz python-26ffbb3c93868dfc85b3933020b7e960bf0bd3a4.tar.bz2 python-26ffbb3c93868dfc85b3933020b7e960bf0bd3a4.zip | |
Imported Upstream version 2.7.16upstream/2.7.16
404 files changed, 9139 insertions, 11427 deletions
diff --git a/Doc/README.txt b/Doc/README.txt index 4f8e9f8..a362ecc 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -15,7 +15,7 @@ Building the docs You need to have Sphinx <http://sphinx-doc.org/> installed; it is the toolset used to build the docs. It is not included in this tree, but maintained -separately and available from PyPI <https://pypi.python.org/pypi/Sphinx>. +separately and available from PyPI <https://pypi.org/project/Sphinx>. Using make diff --git a/Doc/bugs.rst b/Doc/bugs.rst index 25ce3ca..3749f93 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -14,7 +14,7 @@ Documentation bugs If you find a bug in this documentation or would like to propose an improvement, please submit a bug report on the :ref:`tracker <using-the-tracker>`. If you -have a suggestion how to fix it, include that as well. +have a suggestion on how to fix it, include that as well. If you're short on time, you can also email documentation bug reports to docs@python.org (behavioral bugs can be sent to python-list@python.org). @@ -84,5 +84,5 @@ the `core-mentorship mailing list`_ is a friendly place to get answers to any and all questions pertaining to the process of fixing issues in Python. .. _Documentation bugs: https://bugs.python.org/issue?@filter=status&@filter=components&components=4&status=1&@columns=id,activity,title,status&@sort=-activity -.. _Python Developer's Guide: https://docs.python.org/devguide/ -.. _core-mentorship mailing list: https://mail.python.org/mailman/listinfo/core-mentorship/ +.. _Python Developer's Guide: https://devguide.python.org/ +.. _core-mentorship mailing list: https://mail.python.org/mailman3/lists/core-mentorship.python.org/ diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 4e5a043..77940ac 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -98,7 +98,7 @@ The new-style Py_buffer struct suboffset value that it negative indicates that no de-referencing should occur (striding in a contiguous memory block). - If all suboffsets are negative (i.e. no de-referencing is needed, then + If all suboffsets are negative (i.e. no de-referencing is needed), then this field must be NULL (the default value). Here is a function that returns a pointer to the element in an N-D array diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst index b8642d0..fd4279c 100644 --- a/Doc/c-api/capsule.rst +++ b/Doc/c-api/capsule.rst @@ -9,6 +9,8 @@ Capsules Refer to :ref:`using-capsules` for more information on using these objects. +.. versionadded:: 2.7 + .. c:type:: PyCapsule @@ -19,6 +21,7 @@ Refer to :ref:`using-capsules` for more information on using these objects. regular import mechanism can be used to access C APIs defined in dynamically loaded modules. + .. c:type:: PyCapsule_Destructor The type of a destructor callback for a capsule. Defined as:: @@ -104,8 +107,8 @@ Refer to :ref:`using-capsules` for more information on using these objects. import the module conventionally (using :c:func:`PyImport_ImportModule`). Return the capsule's internal *pointer* on success. On failure, set an - exception and return *NULL*. However, if :c:func:`PyCapsule_Import` failed to - import the module, and *no_block* was true, no exception is set. + exception and return *NULL*. + .. c:function:: int PyCapsule_IsValid(PyObject *capsule, const char *name) @@ -122,18 +125,21 @@ Refer to :ref:`using-capsules` for more information on using these objects. Return a nonzero value if the object is valid and matches the name passed in. Return ``0`` otherwise. This function will not fail. + .. c:function:: int PyCapsule_SetContext(PyObject *capsule, void *context) Set the context pointer inside *capsule* to *context*. Return ``0`` on success. Return nonzero and set an exception on failure. + .. c:function:: int PyCapsule_SetDestructor(PyObject *capsule, PyCapsule_Destructor destructor) Set the destructor inside *capsule* to *destructor*. Return ``0`` on success. Return nonzero and set an exception on failure. + .. c:function:: int PyCapsule_SetName(PyObject *capsule, const char *name) Set the name inside *capsule* to *name*. If non-*NULL*, the name must @@ -142,6 +148,7 @@ Refer to :ref:`using-capsules` for more information on using these objects. Return ``0`` on success. Return nonzero and set an exception on failure. + .. c:function:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer) Set the void pointer inside *capsule* to *pointer*. The pointer may not be diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index 9438fea..18ecd07 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -56,7 +56,7 @@ Constructors for container types must conform to two rules: .. c:function:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize) Resize an object allocated by :c:func:`PyObject_NewVar`. Returns the - resized object or *NULL* on failure. + resized object or *NULL* on failure. *op* must not be tracked by the collector yet. .. versionchanged:: 2.5 This function used an :c:type:`int` type for *newsize*. This might diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 258898d..2b9fb73 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -35,7 +35,7 @@ operate within the bounds of the private heap. It is important to understand that the management of the Python heap is performed by the interpreter itself and that the user has no control over it, -even if she regularly manipulates object pointers to memory blocks inside that +even if they regularly manipulate object pointers to memory blocks inside that heap. The allocation of heap space for Python objects and other internal buffers is performed on demand by the Python memory manager through the Python/C API functions listed in this document. diff --git a/Doc/copyright.rst b/Doc/copyright.rst index 540ff5e..393a1f0 100644 --- a/Doc/copyright.rst +++ b/Doc/copyright.rst @@ -4,7 +4,7 @@ Copyright Python and this documentation is: -Copyright © 2001-2018 Python Software Foundation. All rights reserved. +Copyright © 2001-2019 Python Software Foundation. All rights reserved. Copyright © 2000 BeOpen.com. All rights reserved. diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst index 82ecd2c..b0dccb3 100644 --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -31,7 +31,7 @@ installing other Python projects, refer to the Key terms ========= -* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public +* the `Python Packaging Index <https://pypi.org>`__ is a public repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 18844dd..eade905 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -78,7 +78,7 @@ setup script). Indirectly provides the :class:`distutils.dist.Distribution` and | | be built | :class:`distutils.core.Extension` | +--------------------+--------------------------------+-------------------------------------------------------------+ | *classifiers* | A list of categories for the | a list of strings; valid classifiers are listed on `PyPI | - | | package | <https://pypi.python.org/pypi?:action=list_classifiers>`_. | + | | package | <https://pypi.org/classifiers>`_. | +--------------------+--------------------------------+-------------------------------------------------------------+ | *distclass* | the :class:`Distribution` | a subclass of | | | class to use | :class:`distutils.core.Distribution` | @@ -934,7 +934,7 @@ timestamp dependency analysis. .. function:: newer_group(sources, target[, missing='error']) Return true if *target* is out-of-date with respect to any file listed in - *sources* In other words, if *target* exists and is newer than every file in + *sources*. In other words, if *target* exists and is newer than every file in *sources*, return false; otherwise return true. *missing* controls what we do when a source file is missing; the default (``'error'``) is to blow up with an :exc:`OSError` from inside :func:`os.stat`; if it is ``'ignore'``, we silently diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index 15ab3b3..acd499a 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -21,7 +21,7 @@ specialty---writing code and creating source distributions---while an intermediary species called *packagers* springs up to turn source distributions into built distributions for as many platforms as there are packagers. -Of course, the module developer could be his own packager; or the packager could +Of course, the module developer could be their own packager; or the packager could be a volunteer "out there" somewhere who has access to a platform which the original developer does not; or it could be software periodically grabbing new source distributions and turning them into built distributions for as many diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index fc6184f..488619e 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -94,7 +94,7 @@ containing your setup script :file:`setup.py`, and your module :file:`foo.py`. The archive file will be named :file:`foo-1.0.tar.gz` (or :file:`.zip`), and will unpack into a directory :file:`foo-1.0`. -If an end-user wishes to install your :mod:`foo` module, all she has to do is +If an end-user wishes to install your :mod:`foo` module, all they have to do is download :file:`foo-1.0.tar.gz` (or :file:`.zip`), unpack it, and---from the :file:`foo-1.0` directory---run :: @@ -193,8 +193,8 @@ modules using the Distutils: module distribution a collection of Python modules distributed together as a single downloadable resource and meant to be installed *en masse*. Examples of some well-known - module distributions are Numeric Python, PyXML, PIL (the Python Imaging - Library), or mxBase. (This would be called a *package*, except that term is + module distributions are Numeric Python, PyXML, Pillow, + or mxBase. (This would be called a *package*, except that term is already taken in the Python context: a single module distribution may contain zero, one, or many Python packages.) diff --git a/Doc/distutils/packageindex.rst b/Doc/distutils/packageindex.rst index 8087584..cd11f20 100644 --- a/Doc/distutils/packageindex.rst +++ b/Doc/distutils/packageindex.rst @@ -248,4 +248,4 @@ without warnings does not guarantee that PyPI will convert the content successfully. -.. _Python Package Index (PyPI): https://pypi.python.org/pypi +.. _Python Package Index (PyPI): https://pypi.org diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst index 6b82c40..8407206 100644 --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -520,20 +520,23 @@ following way:: setup(..., data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), ('config', ['cfg/data.cfg']), - ('/etc/init.d', ['init-script'])] ) -Note that you can specify the directory names where the data files will be -installed, but you cannot rename the data files themselves. - Each (*directory*, *files*) pair in the sequence specifies the installation -directory and the files to install there. If *directory* is a relative path, it -is interpreted relative to the installation prefix (Python's ``sys.prefix`` for -pure-Python packages, ``sys.exec_prefix`` for packages that contain extension -modules). Each file name in *files* is interpreted relative to the -:file:`setup.py` script at the top of the package source distribution. No -directory information from *files* is used to determine the final location of -the installed file; only the name of the file is used. +directory and the files to install there. + +Each file name in *files* is interpreted relative to the :file:`setup.py` +script at the top of the package source distribution. Note that you can +specify the directory where the data files will be installed, but you cannot +rename the data files themselves. + +The *directory* should be a relative path. It is interpreted relative to the +installation prefix (Python's ``sys.prefix`` for system installations; +``site.USER_BASE`` for user installations). Distutils allows *directory* to be +an absolute installation path, but this is discouraged since it is +incompatible with the wheel packaging format. No directory information from +*files* is used to determine the final location of the installed file; only +the name of the file is used. You can specify the ``data_files`` options as a simple sequence of files without specifying a target directory, but this is not recommended, and the @@ -606,7 +609,7 @@ Notes: (4) These fields should not be used if your package is to be compatible with Python versions prior to 2.2.3 or 2.3. The list is available from the `PyPI website - <https://pypi.python.org/pypi>`_. + <https://pypi.org>`_. (5) The ``long_description`` field is used by PyPI when you are diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 15e4af5..8588daa 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -2,6 +2,11 @@ Design and History FAQ ====================== +.. only:: html + + .. contents:: + + Why does Python use indentation for grouping of statements? ----------------------------------------------------------- @@ -221,24 +226,25 @@ Python file objects support the iterator protocol, so you can now write simply:: Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))? ---------------------------------------------------------------------------------------------------------------- -The major reason is history. Functions were used for those operations that were -generic for a group of types and which were intended to work even for objects -that didn't have methods at all (e.g. tuples). It is also convenient to have a -function that can readily be applied to an amorphous collection of objects when -you use the functional features of Python (``map()``, ``zip()`` et al). +As Guido said: -In fact, implementing ``len()``, ``max()``, ``min()`` as a built-in function is -actually less code than implementing them as methods for each type. One can -quibble about individual cases but it's a part of Python, and it's too late to -make such fundamental changes now. The functions have to remain to avoid massive -code breakage. + (a) For some operations, prefix notation just reads better than + postfix -- prefix (and infix!) operations have a long tradition in + mathematics which likes notations where the visuals help the + mathematician thinking about a problem. Compare the easy with which we + rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of + doing the same thing using a raw OO notation. -.. XXX talk about protocols? + (b) When I read code that says len(x) I *know* that it is asking for + the length of something. This tells me two things: the result is an + integer, and the argument is some kind of container. To the contrary, + when I read x.len(), I have to already know that x is some kind of + container implementing an interface or inheriting from a class that + has a standard len(). Witness the confusion we occasionally have when + a class that is not implementing a mapping has a get() or keys() + method, or something that isn't a file has a write() method. -.. note:: - - For string operations, Python has moved from external functions (the - ``string`` module) to methods. However, ``len()`` is still a function. + -- https://mail.python.org/pipermail/python-3000/2006-November/004643.html Why is join() a string method instead of a list or tuple method? @@ -498,10 +504,10 @@ you can always change a list's elements. Only immutable elements can be used as dictionary keys, and hence only tuples and not lists can be used as keys. -How are lists implemented? --------------------------- +How are lists implemented in CPython? +------------------------------------- -Python's lists are really variable-length arrays, not Lisp-style linked lists. +CPython's lists are really variable-length arrays, not Lisp-style linked lists. The implementation uses a contiguous array of references to other objects, and keeps a pointer to this array and the array's length in a list head structure. @@ -514,10 +520,10 @@ when the array must be grown, some extra space is allocated so the next few times don't require an actual resize. -How are dictionaries implemented? ---------------------------------- +How are dictionaries implemented in CPython? +-------------------------------------------- -Python's dictionaries are implemented as resizable hash tables. Compared to +CPython's dictionaries are implemented as resizable hash tables. Compared to B-trees, this gives better performance for lookup (the most common operation by far) under most circumstances, and the implementation is simpler. diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst index 4be58d6..0a256b5 100644 --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -286,7 +286,7 @@ However sometimes you have to run the embedded Python interpreter in the same thread as your rest application and you can't allow the :c:func:`PyRun_InteractiveLoop` to stop while waiting for user input. The one solution then is to call :c:func:`PyParser_ParseString` and test for ``e.error`` -equal to ``E_EOF``, which means the input is incomplete). Here's a sample code +equal to ``E_EOF``, which means the input is incomplete. Here's a sample code fragment, untested, inspired by code from Alex Farber:: #include <Python.h> diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst index 35cad55..887dea7 100644 --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -117,7 +117,7 @@ programming), software engineering (unit testing, logging, profiling, parsing Python code), and operating system interfaces (system calls, filesystems, TCP/IP sockets). Look at the table of contents for :ref:`library-index` to get an idea of what's available. A wide variety of third-party extensions are also -available. Consult `the Python Package Index <https://pypi.python.org/pypi>`_ to +available. Consult `the Python Package Index <https://pypi.org>`_ to find packages of interest to you. @@ -306,17 +306,19 @@ usually around 18 months between major releases. The developers issue "bugfix" releases of older versions, so the stability of existing releases gradually improves. Bugfix releases, indicated by a third -component of the version number (e.g. 2.5.3, 2.6.2), are managed for stability; +component of the version number (e.g. 3.5.3, 3.6.2), are managed for stability; only fixes for known problems are included in a bugfix release, and it's guaranteed that interfaces will remain the same throughout a series of bugfix releases. The latest stable releases can always be found on the `Python download page -<https://www.python.org/downloads/>`_. There are two recommended production-ready -versions at this point in time, because at the moment there are two branches of -stable releases: 2.x and 3.x. Python 3.x may be less useful than 2.x, since -currently there is more third party software available for Python 2 than for -Python 3. Python 2 code will generally not run unchanged in Python 3. +<https://www.python.org/downloads/>`_. There are two production-ready version +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x. +Although Python 2.x is still widely used, `it will not be +maintained after January 1, 2020 <https://www.python.org/dev/peps/pep-0373/>`_. +Python 2.x was known for having more third-party libraries available, however, +by the time of this writing, most of the widely used libraries support Python 3.x, +and some are even dropping the Python 2.x support. How many people are using Python? diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index d1b3efb..a259465 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -19,7 +19,7 @@ standard library module. (Eventually you'll learn what's in the standard library and will be able to skip this step.) For third-party packages, search the `Python Package Index -<https://pypi.python.org/pypi>`_ or try `Google <https://www.google.com>`_ or +<https://pypi.org>`_ or try `Google <https://www.google.com>`_ or another Web search engine. Searching for "Python" plus a keyword or two for your topic of interest will usually find something helpful. @@ -585,7 +585,7 @@ substituted for standard input and output. You will have to use pseudo ttys ("ptys") instead of pipes. Or you can use a Python interface to Don Libes' "expect" library. A Python extension that interfaces to expect is called "expy" and available from http://expectpy.sourceforge.net. A pure Python solution that -works like expect is `pexpect <https://pypi.python.org/pypi/pexpect/>`_. +works like expect is `pexpect <https://pypi.org/project/pexpect/>`_. How do I access the serial (RS232) port? diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst index f8e23cf..7f0c955 100644 --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -17,18 +17,6 @@ This is not necessarily a straightforward question. If you are already familiar with running programs from the Windows command line then everything will seem obvious; otherwise, you might need a little more guidance. -.. sidebar:: |Python Development on XP|_ - :subtitle: `Python Development on XP`_ - - This series of screencasts aims to get you up and running with Python on - Windows XP. The knowledge is distilled into 1.5 hours and will get you up - and running with the right Python distribution, coding in your choice of IDE, - and debugging and writing solid code with unit-tests. - -.. |Python Development on XP| image:: python-video-icon.png -.. _`Python Development on XP`: - http://showmedo.com/videotutorials/series?name=pythonOzsvaldPyNewbieSeries - Unless you use some sort of integrated development environment, you will end up *typing* Windows commands into what is variously referred to as a "DOS window" or "Command prompt window". Usually you can create such a window from your @@ -92,18 +80,6 @@ gives you a message like:: 'python' is not recognized as an internal or external command, operable program or batch file. -.. sidebar:: |Adding Python to DOS Path|_ - :subtitle: `Adding Python to DOS Path`_ - - Python is not added to the DOS path by default. This screencast will walk - you through the steps to add the correct entry to the `System Path`, allowing - Python to be executed from the command-line by all users. - -.. |Adding Python to DOS Path| image:: python-video-icon.png -.. _`Adding Python to DOS Path`: - http://showmedo.com/videotutorials/video?name=960000&fromSeriesID=96 - - or:: Bad command or filename diff --git a/Doc/glossary.rst b/Doc/glossary.rst index cbd1237..9e6bf23 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -14,8 +14,9 @@ Glossary ``...`` The default Python prompt of the interactive shell when entering code for - an indented code block or within a pair of matching left and right - delimiters (parentheses, square brackets or curly braces). + an indented code block, when within a pair of matching left and right + delimiters (parentheses, square brackets, curly braces or triple quotes), + or after specifying a decorator. 2to3 A tool that tries to convert Python 2.x code to Python 3.x code by @@ -296,7 +297,7 @@ Glossary the :func:`next` function. Each :keyword:`yield` temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator resumes, it - picks-up where it left-off (in contrast to functions which start fresh on + picks up where it left off (in contrast to functions which start fresh on every invocation). .. index:: single: generator expression @@ -457,7 +458,7 @@ Glossary lambda An anonymous inline function consisting of a single :term:`expression` which is evaluated when the function is called. The syntax to create - a lambda function is ``lambda [arguments]: expression`` + a lambda function is ``lambda [parameters]: expression`` LBYL Look before you leap. This coding style explicitly tests for @@ -474,7 +475,7 @@ Glossary list A built-in Python :term:`sequence`. Despite its name it is more akin to an array in other languages than to a linked list since access to - elements are O(1). + elements is O(1). list comprehension A compact way to process all or part of the elements in a sequence and @@ -625,6 +626,21 @@ Glossary :ref:`the difference between arguments and parameters <faq-argument-vs-parameter>`, and the :ref:`function` section. + PEP + Python Enhancement Proposal. A PEP is a design document + providing information to the Python community, or describing a new + feature for Python or its processes or environment. PEPs should + provide a concise technical specification and a rationale for proposed + features. + + PEPs are intended to be the primary mechanisms for proposing major new + features, for collecting community input on an issue, and for documenting + the design decisions that have gone into Python. The PEP author is + responsible for building consensus within the community and documenting + dissenting opinions. + + See :pep:`1`. + positional argument See :term:`argument`. @@ -694,7 +710,7 @@ Glossary struct sequence A tuple with named elements. Struct sequences expose an interface similiar - to :term:`named tuple` in that elements can either be accessed either by + to :term:`named tuple` in that elements can be accessed either by index or as an attribute. However, they do not have any of the named tuple methods like :meth:`~collections.somenamedtuple._make` or :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences diff --git a/Doc/howto/doanddont.rst b/Doc/howto/doanddont.rst index 98dbad1..35e1583 100644 --- a/Doc/howto/doanddont.rst +++ b/Doc/howto/doanddont.rst @@ -285,7 +285,7 @@ There are also many useful built-in functions people seem not to be aware of for some reason: :func:`min` and :func:`max` can find the minimum/maximum of any sequence with comparable semantics, for example, yet many people write their own :func:`max`/:func:`min`. Another highly useful function is -:func:`reduce` which can be used to repeatly apply a binary operation to a +:func:`reduce` which can be used to repeatedly apply a binary operation to a sequence, reducing it to a single value. For example, compute a factorial with a series of multiply operations:: diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 8562d23..88b0177 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -427,25 +427,25 @@ to make sure everything functions as expected in both versions of Python. .. _2to3: https://docs.python.org/3/library/2to3.html -.. _caniusepython3: https://pypi.python.org/pypi/caniusepython3 +.. _caniusepython3: https://pypi.org/project/caniusepython3 .. _cheat sheet: http://python-future.org/compatible_idioms.html -.. _coverage.py: https://pypi.python.org/pypi/coverage +.. _coverage.py: https://pypi.org/project/coverage .. _Futurize: http://python-future.org/automatic_conversion.html .. _importlib: https://docs.python.org/3/library/importlib.html#module-importlib -.. _importlib2: https://pypi.python.org/pypi/importlib2 +.. _importlib2: https://pypi.org/project/importlib2 .. _Modernize: https://python-modernize.readthedocs.org/en/latest/ .. _mypy: http://mypy-lang.org/ .. _Porting to Python 3: http://python3porting.com/ -.. _Pylint: https://pypi.python.org/pypi/pylint +.. _Pylint: https://pypi.org/project/pylint .. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html .. _pytype: https://github.com/google/pytype .. _python-future: http://python-future.org/ .. _python-porting: https://mail.python.org/mailman/listinfo/python-porting -.. _six: https://pypi.python.org/pypi/six -.. _tox: https://pypi.python.org/pypi/tox -.. _trove classifier: https://pypi.python.org/pypi?%3Aaction=list_classifiers +.. _six: https://pypi.org/project/six +.. _tox: https://pypi.org/project/tox +.. _trove classifier: https://pypi.org/classifiers .. _"What's New": https://docs.python.org/3/whatsnew/index.html diff --git a/Doc/howto/webservers.rst b/Doc/howto/webservers.rst index a555083..5071a8a 100644 --- a/Doc/howto/webservers.rst +++ b/Doc/howto/webservers.rst @@ -301,7 +301,7 @@ following WSGI-application:: WSGIServer(app).run() This is a simple WSGI application, but you need to install `flup -<https://pypi.python.org/pypi/flup/1.0>`_ first, as flup handles the low level +<https://pypi.org/project/flup/1.0>`_ first, as flup handles the low level FastCGI access. .. seealso:: @@ -583,7 +583,7 @@ alternate storage mechanism. helps with choosing a method for saving data * `SQLAlchemy <http://www.sqlalchemy.org/>`_, the most powerful OR-Mapper - for Python, and `Elixir <https://pypi.python.org/pypi/Elixir>`_, which makes + for Python, and `Elixir <https://pypi.org/project/Elixir>`_, which makes SQLAlchemy easier to use * `SQLObject <http://www.sqlobject.org/>`_, another popular OR-Mapper diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 5a7df0d..8276b01 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -3,7 +3,7 @@ int main(int argc, char *argv[]) { - PyObject *pName, *pModule, *pDict, *pFunc; + PyObject *pName, *pModule, *pFunc; PyObject *pArgs, *pValue; int i; diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index cbbb882..9b89d08 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -39,9 +39,8 @@ Key terms being installed system wide * ``virtualenv`` is a third party tools for creating virtual environments, it is defaults to installing ``pip`` into all created virtual environments. -* the `Python Packaging Index <https://pypi.python.org/pypi>`__ is a public - repository of open source licensed packages made available for use by - other Python users +* the `Python Packaging Index <https://pypi.org>`__ is a public repository of + open source licensed packages made available for use by other Python users * the `Python Packaging Authority <https://www.pypa.io/en/latest/>`__ are the group of developers and documentation authors responsible for the maintenance and diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index 3a00a9c..b5ba139 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -19,10 +19,10 @@ was added to Python. (See also :pep:`3141` and the :mod:`numbers` module regarding a type hierarchy for numbers based on ABCs.) The :mod:`collections` module has some concrete classes that derive from -ABCs; these can, of course, be further derived. In addition the +ABCs; these can, of course, be further derived. In addition, the :mod:`collections` module has some ABCs that can be used to test whether -a class or instance provides a particular interface, for example, is it -hashable or a mapping. +a class or instance provides a particular interface, for example, if it is +hashable or if it is a mapping. This module provides the following class: diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 71957ec..478a842 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -35,6 +35,10 @@ Here is a summary of the features offered by the bz2 module: * Thread safety uses individual locking mechanism. +.. note:: + Handling of multi-stream bzip2 files is not supported. Modules such as + `bz2file <https://github.com/nvawda/bz2file>`_ let you overcome this. + (De)compression of files ------------------------ @@ -74,7 +78,7 @@ Handling of compressed files is offered by the :class:`BZ2File` class. input file, only the first stream will be accessible. If you require support for multi-stream files, consider using the third-party :mod:`bz2file` module (available from - `PyPI <https://pypi.python.org/pypi/bz2file>`_). This module provides a + `PyPI <https://pypi.org/project/bz2file>`_). This module provides a backport of Python 3.3's :class:`BZ2File` class, which does support multi-stream files. diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 1bfdb39..ecd62c8 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -292,12 +292,12 @@ algorithms implemented in this module in other circumstances. passed to :func:`urlparse.parse_qs` unchanged. -.. function:: parse_qs(qs[, keep_blank_values[, strict_parsing]]) +.. function:: parse_qs(qs[, keep_blank_values[, strict_parsing[, max_num_fields]]]) This function is deprecated in this module. Use :func:`urlparse.parse_qs` instead. It is maintained here only for backward compatibility. -.. function:: parse_qsl(qs[, keep_blank_values[, strict_parsing]]) +.. function:: parse_qsl(qs[, keep_blank_values[, strict_parsing[, max_num_fields]]]) This function is deprecated in this module. Use :func:`urlparse.parse_qsl` instead. It is maintained here only for backward compatibility. diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index c88dd23..183da70 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -24,22 +24,28 @@ Functions provided: function for :keyword:`with` statement context managers, without needing to create a class or separate :meth:`__enter__` and :meth:`__exit__` methods. - A simple example (this is not recommended as a real way of generating HTML!):: + While many objects natively support use in with statements, sometimes a + resource needs to be managed that isn't a context manager in its own right, + and doesn't implement a ``close()`` method for use with ``contextlib.closing`` + + An abstract example would be the following to ensure correct resource + management:: from contextlib import contextmanager @contextmanager - def tag(name): - print "<%s>" % name - yield - print "</%s>" % name - - >>> with tag("h1"): - ... print "foo" - ... - <h1> - foo - </h1> + def managed_resource(*args, **kwds): + # Code to acquire resource, e.g.: + resource = acquire_resource(*args, **kwds) + try: + yield resource + finally: + # Code to release resource, e.g.: + release_resource(resource) + + >>> with managed_resource(timeout=3600) as resource: + ... # Resource is released at the end of this block, + ... # even if code in the block raises an exception The function being decorated must return a :term:`generator`-iterator when called. This iterator must yield exactly one value, which will be bound to diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index fedd370..ad6b716 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -453,8 +453,9 @@ read CSV files (assuming they support complex numbers at all). .. method:: csvwriter.writerows(rows) - Write all the *rows* parameters (a list of *row* objects as described above) to - the writer's file object, formatted according to the current dialect. + Write all elements in *rows* (an iterable of *row* objects as described + above) to the writer's file object, formatted according to the current + dialect. Writer objects have the following public attribute: diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index b8d86e3..2d164b2 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -1566,7 +1566,7 @@ EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). .. seealso:: - `pytz <https://pypi.python.org/pypi/pytz/>`_ + `pytz <https://pypi.org/project/pytz/>`_ The standard library has no :class:`tzinfo` instances, but there exists a third-party library which brings the *IANA timezone database* (also known as the Olson database) to Python: *pytz*. @@ -1594,7 +1594,9 @@ although not all objects support a :meth:`timetuple` method. Conversely, the :meth:`datetime.strptime` class method creates a :class:`.datetime` object from a string representing a date and time and a corresponding format string. ``datetime.strptime(date_string, format)`` is -equivalent to ``datetime(*(time.strptime(date_string, format)[0:6]))``. +equivalent to ``datetime(*(time.strptime(date_string, format)[0:6]))``, except +when the format includes sub-second components or timezone offset information, +which are supported in ``datetime.strptime`` but are discarded by ``time.strptime``. For :class:`.time` objects, the format codes for year, month, and day should not be used, as time objects have no such values. If they're used anyway, ``1900`` @@ -1609,6 +1611,12 @@ calls the platform C library's :func:`strftime` function, and platform variations are common. To see the full set of format codes supported on your platform, consult the :manpage:`strftime(3)` documentation. +For the same reason, handling of format strings containing Unicode code points +that can't be represented in the charset of the current locale is also +platform-dependent. On some platforms such code points are preserved intact in +the output, while on others ``strftime`` may raise :exc:`UnicodeError` or return +an empty string instead. + The following is a list of all the format codes that the C standard (1989 version) requires, and these work on all platforms with a standard C implementation. Note that the 1999 version of the C standard added additional diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index c6bf3ef..7b6cce4 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -431,14 +431,15 @@ The :class:`SequenceMatcher` class has this constructor: .. method:: get_matching_blocks() - Return list of triples describing matching subsequences. Each triple is of - the form ``(i, j, n)``, and means that ``a[i:i+n] == b[j:j+n]``. The + Return list of triples describing non-overlapping matching subsequences. + Each triple is of the form ``(i, j, n)``, + and means that ``a[i:i+n] == b[j:j+n]``. The triples are monotonically increasing in *i* and *j*. The last triple is a dummy, and has the value ``(len(a), len(b), 0)``. It is the only triple with ``n == 0``. If ``(i, j, n)`` and ``(i', j', n')`` are adjacent triples in the list, and the second is not the last triple in - the list, then ``i+n != i'`` or ``j+n != j'``; in other words, adjacent + the list, then ``i+n < i'`` or ``j+n < j'``; in other words, adjacent triples always describe non-adjacent equal blocks. .. XXX Explain why a dummy is used! @@ -757,8 +758,10 @@ It is also contained in the Python source distribution, as # we're passing these as arguments to the diff function fromdate = time.ctime(os.stat(fromfile).st_mtime) todate = time.ctime(os.stat(tofile).st_mtime) - fromlines = open(fromfile, 'U').readlines() - tolines = open(tofile, 'U').readlines() + with open(fromfile, 'U') as f: + fromlines = f.readlines() + with open(tofile, 'U') as f: + tolines = f.readlines() if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 04b9b15..e773ab8 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -107,7 +107,7 @@ The :mod:`dis` module defines the following functions and constants: .. data:: hasconst - Sequence of bytecodes that have a constant parameter. + Sequence of bytecodes that access a constant. .. data:: hasfree @@ -796,21 +796,25 @@ the more significant byte last. .. opcode:: RAISE_VARARGS (argc) - Raises an exception. *argc* indicates the number of parameters to the raise + Raises an exception. *argc* indicates the number of arguments to the raise statement, ranging from 0 to 3. The handler will find the traceback as TOS2, the parameter as TOS1, and the exception as TOS. .. opcode:: CALL_FUNCTION (argc) - Calls a function. The low byte of *argc* indicates the number of positional - parameters, the high byte the number of keyword parameters. On the stack, the - opcode finds the keyword parameters first. For each keyword argument, the - value is on top of the key. Below the keyword parameters, the positional - parameters are on the stack, with the right-most parameter on top. Below the - parameters, the function object to call is on the stack. Pops all function - arguments, and the function itself off the stack, and pushes the return - value. + Calls a callable object. The low byte of *argc* indicates the number of + positional arguments, the high byte the number of keyword arguments. + The stack contains keyword arguments on top (if any), then the positional + arguments below that (if any), then the callable object to call below that. + Each keyword argument is represented with two values on the stack: + the argument's name, and its value, with the argument's value above the + name on the stack. + The positional arguments are pushed in the order that they are passed in + to the callable object, with the right-most positional argument on top. + ``CALL_FUNCTION`` pops all arguments and the callable object off the stack, + calls the callable object with those arguments, and pushes the return value + returned by the callable object. .. opcode:: MAKE_FUNCTION (argc) @@ -847,24 +851,53 @@ the more significant byte last. .. opcode:: CALL_FUNCTION_VAR (argc) - Calls a function. *argc* is interpreted as in :opcode:`CALL_FUNCTION`. The - top element on the stack contains the variable argument list, followed by - keyword and positional arguments. + Calls a callable object, similarly to :opcode:`CALL_FUNCTION`. + *argc* represents the number of keyword and positional + arguments, identically to :opcode:`CALL_FUNCTION`. + The top of the stack contains an iterable object containing + additional positional arguments. + Below that are keyword arguments (if any), positional arguments (if any) + and a callable object, identically to :opcode:`CALL_FUNCTION`. + Before the callable object is called, the iterable object + is "unpacked" and its contents are appended to the positional + arguments passed in. + The iterable object is ignored when computing + the value of ``argc``. .. opcode:: CALL_FUNCTION_KW (argc) - Calls a function. *argc* is interpreted as in :opcode:`CALL_FUNCTION`. The - top element on the stack contains the keyword arguments dictionary, followed - by explicit keyword and positional arguments. + Calls a callable object, similarly to :opcode:`CALL_FUNCTION`. + *argc* represents the number of keyword and positional + arguments, identically to :opcode:`CALL_FUNCTION`. + The top of the stack contains a mapping object containing additional keyword + arguments. + Below that are keyword arguments (if any), positional arguments (if any) + and a callable object, identically to :opcode:`CALL_FUNCTION`. + Before the callable is called, the mapping object at the top of the stack is + "unpacked" and its contents are appended to the keyword arguments passed in. + The mapping object at the top of the stack is ignored when computing + the value of ``argc``. .. opcode:: CALL_FUNCTION_VAR_KW (argc) - Calls a function. *argc* is interpreted as in :opcode:`CALL_FUNCTION`. The - top element on the stack contains the keyword arguments dictionary, followed - by the variable-arguments tuple, followed by explicit keyword and positional - arguments. + Calls a callable object, similarly to :opcode:`CALL_FUNCTION_VAR` and + :opcode:`CALL_FUNCTION_KW`. + *argc* represents the number of keyword and positional + arguments, identically to :opcode:`CALL_FUNCTION`. + The top of the stack contains a mapping object, as per + :opcode:`CALL_FUNCTION_KW`. + Below that is an iterable object, as per + :opcode:`CALL_FUNCTION_VAR`. + Below that are keyword arguments (if any), positional arguments (if any) + and a callable object, identically to :opcode:`CALL_FUNCTION`. + Before the callable is called, the mapping object and iterable object + are each "unpacked" and their contents passed in as keyword and + positional arguments respectively, + identically to :opcode:`CALL_FUNCTION_VAR` and :opcode:`CALL_FUNCTION_KW`. + The mapping object and iterable object are both ignored when computing + the value of ``argc``. .. opcode:: HAVE_ARGUMENT () diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index b560999..21dd18d 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -4,7 +4,7 @@ Software Packaging and Distribution These libraries help you with publishing and installing Python software. While these modules are designed to work in conjunction with the -`Python Package Index <https://pypi.python.org>`__, they can also be used +`Python Package Index <https://pypi.org>`__, they can also be used with a local index server, or without any index server at all. .. toctree:: diff --git a/Doc/library/email.rst b/Doc/library/email.rst index 879c38f..d9bd9ed 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -61,7 +61,7 @@ Contents of the :mod:`email` package documentation: email.charset.rst email.encoders.rst email.errors.rst - email.util.rst + email.utils.rst email.iterators.rst email-examples.rst diff --git a/Doc/library/email.util.rst b/Doc/library/email.utils.rst index 9b583c0..9b583c0 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.utils.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index fe14656..8757c6c 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -439,7 +439,7 @@ The following exceptions are the exceptions that are actually raised. .. exception:: ValueError - Raised when a built-in operation or function receives an argument that has the + Raised when an operation or function receives an argument that has the right type but an inappropriate value, and the situation is not described by a more precise exception such as :exc:`IndexError`. diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index 42bbf74..8618f5f 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -36,7 +36,7 @@ For example, ``'[?]'`` matches the character ``'?'``. Note that the filename separator (``'/'`` on Unix) is *not* special to this module. See module :mod:`glob` for pathname expansion (:mod:`glob` uses -:func:`fnmatch` to match pathname segments). Similarly, filenames starting with +:func:`.filter` to match pathname segments). Similarly, filenames starting with a period are not special for this module, and are matched by the ``*`` and ``?`` patterns. diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index f3e396b..aee14d5 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -76,9 +76,9 @@ The :mod:`functools` module defines the following functions: .. function:: partial(func[,*args][, **keywords]) - Return a new :class:`partial` object which when called will behave like *func* - called with the positional arguments *args* and keyword arguments *keywords*. If - more arguments are supplied to the call, they are appended to *args*. If + Return a new :ref:`partial object<partial-objects>` which when called will behave + like *func* called with the positional arguments *args* and keyword arguments *keywords*. + If more arguments are supplied to the call, they are appended to *args*. If additional keyword arguments are supplied, they extend and override *keywords*. Roughly equivalent to:: diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index a461e51..3848231 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -442,7 +442,7 @@ longer or disable the extension. Calltips ^^^^^^^^ -A calltip is shown when one types :kbd:`(` after the name of an *acccessible* +A calltip is shown when one types :kbd:`(` after the name of an *accessible* function. A name expression may include dots and subscripts. A calltip remains until it is clicked, the cursor is moved out of the argument area, or :kbd:`)` is typed. When the cursor is in the argument part of a definition, diff --git a/Doc/library/index.rst b/Doc/library/index.rst index 97cf3ea..31ca6df 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -30,7 +30,7 @@ optional components. In addition to the standard library, there is a growing collection of several thousand components (from individual programs and modules to packages and entire application development frameworks), available from -the `Python Package Index <https://pypi.python.org/pypi>`_. +the `Python Package Index <https://pypi.org>`_. .. toctree:: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index b99324c..dcdd01c 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -724,7 +724,7 @@ Text I/O .. versionadded:: 2.7 - .. method:: read(n) + .. method:: read(n=-1) Read and return at most *n* characters from the stream as a single :class:`unicode`. If *n* is negative or ``None``, reads until EOF. @@ -809,7 +809,7 @@ Text I/O the given string. If *line_buffering* is ``True``, :meth:`flush` is implied when a call to - write contains a newline character. + write contains a newline character or a carriage return. :class:`TextIOWrapper` provides one attribute in addition to those of :class:`TextIOBase` and its parents: diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 0f568aa..09cae18 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -949,26 +949,26 @@ functions. +--------------+---------------------------------------------+ | Format | Description | +==============+=============================================+ - | ``filename`` | Specifies that a FileHandler be created, | + | *filename* | Specifies that a FileHandler be created, | | | using the specified filename, rather than a | | | StreamHandler. | +--------------+---------------------------------------------+ - | ``filemode`` | Specifies the mode to open the file, if | - | | filename is specified (if filemode is | - | | unspecified, it defaults to 'a'). | + | *filemode* | If *filename* is specified, open the file | + | | in this mode. Defaults to ``'a'``. | +--------------+---------------------------------------------+ - | ``format`` | Use the specified format string for the | + | *format* | Use the specified format string for the | | | handler. | +--------------+---------------------------------------------+ - | ``datefmt`` | Use the specified date/time format. | + | *datefmt* | Use the specified date/time format, as | + | | accepted by :func:`time.strftime`. | +--------------+---------------------------------------------+ - | ``level`` | Set the root logger level to the specified | - | | level. | + | *level* | Set the root logger level to the specified | + | | :ref:`level <levels>`. | +--------------+---------------------------------------------+ - | ``stream`` | Use the specified stream to initialize the | + | *stream* | Use the specified stream to initialize the | | | StreamHandler. Note that this argument is | - | | incompatible with 'filename' - if both are | - | | present, 'stream' is ignored. | + | | incompatible with *filename* - if both are | + | | present, *stream* is ignored. | +--------------+---------------------------------------------+ diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index a86eb13..c9fc733 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -68,7 +68,7 @@ memory but does not update the underlying file. *offset* may be specified as a non-negative integer offset. mmap references will be relative to the offset from the beginning of the file. *offset* - defaults to 0. *offset* must be a multiple of the ALLOCATIONGRANULARITY. + defaults to 0. *offset* must be a multiple of the :const:`ALLOCATIONGRANULARITY`. .. class:: mmap(fileno, length[, flags[, prot[, access[, offset]]]]) @@ -97,8 +97,8 @@ memory but does not update the underlying file. *offset* may be specified as a non-negative integer offset. mmap references will be relative to the offset from the beginning of the file. *offset* - defaults to 0. *offset* must be a multiple of the PAGESIZE or - ALLOCATIONGRANULARITY. + defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY` + which is equal to :const:`PAGESIZE` on Unix systems. To ensure validity of the created memory mapping the file specified by the descriptor *fileno* is internally automatically synchronized @@ -171,7 +171,8 @@ memory but does not update the underlying file. use of this call there is no guarantee that changes are written back before the object is destroyed. If *offset* and *size* are specified, only changes to the given range of bytes will be flushed to disk; otherwise, the - whole extent of the mapping is flushed. + whole extent of the mapping is flushed. *offset* must be a multiple of the + :const:`PAGESIZE` or :const:`ALLOCATIONGRANULARITY`. **(Windows version)** A nonzero value returned indicates success; zero indicates failure. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 97a7b53..2d8660a 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -803,10 +803,13 @@ Miscellaneous Connection Objects ~~~~~~~~~~~~~~~~~~ +.. currentmodule:: None + Connection objects allow the sending and receiving of picklable objects or strings. They can be thought of as message oriented connected sockets. -Connection objects are usually created using :func:`Pipe` -- see also +Connection objects are usually created using +:func:`Pipe <multiprocessing.Pipe>` -- see also :ref:`multiprocessing-listeners-clients`. .. class:: Connection @@ -926,6 +929,8 @@ For example: Synchronization primitives ~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. currentmodule:: multiprocessing + Generally synchronization primitives are not as necessary in a multiprocess program as they are in a multithreaded program. See the documentation for :mod:`threading` module. @@ -1943,8 +1948,7 @@ Listeners and Clients :synopsis: API for dealing with sockets. Usually message passing between processes is done using queues or by using -:class:`~multiprocessing.Connection` objects returned by -:func:`~multiprocessing.Pipe`. +:class:`Connection` objects returned by :func:`~multiprocessing.Pipe`. However, the :mod:`multiprocessing.connection` module allows some extra flexibility. It basically gives a high level message oriented API for dealing @@ -1972,7 +1976,7 @@ authentication* using the :mod:`hmac` module. .. function:: Client(address[, family[, authenticate[, authkey]]]) Attempt to set up a connection to the listener which is using address - *address*, returning a :class:`~multiprocessing.Connection`. + *address*, returning a :class:`Connection`. The type of the connection is determined by *family* argument, but this can generally be omitted since it can usually be inferred from the format of @@ -2028,8 +2032,8 @@ authentication* using the :mod:`hmac` module. .. method:: accept() Accept a connection on the bound socket or named pipe of the listener - object and return a :class:`~multiprocessing.Connection` object. If - authentication is attempted and fails, then + object and return a :class:`Connection` object. + If authentication is attempted and fails, then :exc:`~multiprocessing.AuthenticationError` is raised. .. method:: close() @@ -2050,11 +2054,24 @@ authentication* using the :mod:`hmac` module. unavailable then it is ``None``. -The module defines two exceptions: +The module defines the following exceptions: + +.. exception:: ProcessError + + The base class of all :mod:`multiprocessing` exceptions. + +.. exception:: BufferTooShort + + Exception raised by :meth:`Connection.recv_bytes_into()` when the supplied + buffer object is too small for the message read. .. exception:: AuthenticationError - Exception raised when there is an authentication error. + Raised when there is an authentication error. + +.. exception:: TimeoutError + + Raised by methods with a timeout when the timeout expires. **Examples** @@ -2126,7 +2143,7 @@ an ``'AF_PIPE'`` address rather than an ``'AF_UNIX'`` address. Authentication keys ~~~~~~~~~~~~~~~~~~~ -When one uses :meth:`Connection.recv <multiprocessing.Connection.recv>`, the +When one uses :meth:`Connection.recv`, the data received is automatically unpickled. Unfortunately unpickling data from an untrusted source is a security risk. Therefore :class:`Listener` and :func:`Client` use the :mod:`hmac` module diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst index 4af75a1..627eb7d 100644 --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -1679,7 +1679,7 @@ The callback function should raise :exc:`OptionValueError` if there are any problems with the option or its argument(s). :mod:`optparse` catches this and terminates the program, printing the error message you supply to stderr. Your message should be clear, concise, accurate, and mention the option at fault. -Otherwise, the user will have a hard time figuring out what he did wrong. +Otherwise, the user will have a hard time figuring out what they did wrong. .. _optparse-callback-example-1: diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 9f64095..3d0743b 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -270,6 +270,10 @@ Unix Platforms parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. + .. note:: + This function is deprecated since Python 3.5 and removed in Python 3.8. + See alternative like the `distro <https://pypi.org/project/distro>`_ package. + .. versionadded:: 2.6 .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048) diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index ffa27d4..3f09b21 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -234,4 +234,3 @@ parameters. ['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'], ['cccccccccccccccccccc', 'dddddddddddddddddddd']] - diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 210e99c..0926373 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -333,11 +333,12 @@ Analysis of the profiler data is done using the :class:`~pstats.Stats` class. corresponding version of :mod:`profile` or :mod:`cProfile`. To be specific, there is *no* file compatibility guaranteed with future versions of this profiler, and there is no compatibility with files produced by other - profilers. If several files are provided, all the statistics for identical - functions will be coalesced, so that an overall view of several processes can - be considered in a single report. If additional files need to be combined - with data in an existing :class:`~pstats.Stats` object, the - :meth:`~pstats.Stats.add` method can be used. + profilers, or the same profiler run on a different operating system. If + several files are provided, all the statistics for identical functions will + be coalesced, so that an overall view of several processes can be considered + in a single report. If additional files need to be combined with data in an + existing :class:`~pstats.Stats` object, the :meth:`~pstats.Stats.add` method + can be used. Instead of reading the profile data from a file, a :class:`cProfile.Profile` or :class:`profile.Profile` object can be used as the profile data source. diff --git a/Doc/library/re.rst b/Doc/library/re.rst index c424230..c708a29 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -35,7 +35,7 @@ fine-tuning parameters. .. seealso:: - The third-party `regex <https://pypi.python.org/pypi/regex/>`_ module, + The third-party `regex <https://pypi.org/project/regex/>`_ module, which has an API compatible with the standard library :mod:`re` module, but offers additional functionality and a more thorough Unicode support. @@ -1256,8 +1256,8 @@ Finding all Adverbs ^^^^^^^^^^^^^^^^^^^ :func:`findall` matches *all* occurrences of a pattern, not just the first -one as :func:`search` does. For example, if one was a writer and wanted to -find all of the adverbs in some text, he or she might use :func:`findall` in +one as :func:`search` does. For example, if a writer wanted to +find all of the adverbs in some text, they might use :func:`findall` in the following manner: >>> text = "He was carefully disguised but captured quickly by police." @@ -1271,8 +1271,8 @@ Finding all Adverbs and their Positions If one wants more information about all matches of a pattern than the matched text, :func:`finditer` is useful as it provides instances of :class:`MatchObject` instead of strings. Continuing with the previous example, -if one was a writer who wanted to find all of the adverbs *and their positions* -in some text, he or she would use :func:`finditer` in the following manner: +if a writer wanted to find all of the adverbs *and their positions* +in some text, they would use :func:`finditer` in the following manner: >>> text = "He was carefully disguised but captured quickly by police." >>> for m in re.finditer(r"\w+ly", text): diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 6bd8f05..7e9af1a 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -82,9 +82,11 @@ Directory and files operations .. function:: copy2(src, dst) - Similar to :func:`shutil.copy`, but metadata is copied as well -- in fact, - this is just :func:`shutil.copy` followed by :func:`copystat`. This is - similar to the Unix command :program:`cp -p`. + Identical to :func:`~shutil.copy` except that :func:`copy2` + also attempts to preserve file metadata. + + :func:`copy2` uses :func:`copystat` to copy the file metadata. + Please see :func:`copystat` for more information. .. function:: ignore_patterns(\*patterns) diff --git a/Doc/library/simplehttpserver.rst b/Doc/library/simplehttpserver.rst index 2e7e97a..df8699e 100644 --- a/Doc/library/simplehttpserver.rst +++ b/Doc/library/simplehttpserver.rst @@ -11,6 +11,10 @@ Python 3. The :term:`2to3` tool will automatically adapt imports when converting your sources to Python 3. +.. warning:: + + mod:`SimpleHTTServer` is not recommended for production. It only implements + basic security checks. The :mod:`SimpleHTTPServer` module defines a single class, :class:`SimpleHTTPRequestHandler`, which is interface-compatible with diff --git a/Doc/library/site.rst b/Doc/library/site.rst index ff7195d..1c2d63e 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -206,7 +206,7 @@ If it is called without arguments, it will print the contents of If both options are given, user base and user site will be printed (always in this order), separated by :data:`os.pathsep`. -If any option is given, the script will exit with one of these values: ``O`` if +If any option is given, the script will exit with one of these values: ``0`` if the user site-packages directory is enabled, ``1`` if it was disabled by the user, ``2`` if it is disabled for security reasons or by an administrator, and a value greater than 2 if there is an error. diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index b6e7689..2cc519e 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -36,7 +36,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). is raised. For normal use, you should only require the initialization/connect, - :meth:`sendmail`, and :meth:`~smtplib.quit` methods. + :meth:`sendmail`, and :meth:`SMTP.quit` methods. An example is included below. .. versionchanged:: 2.6 diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index bd5dd14..ce53e84 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -206,8 +206,8 @@ Module functions and constants Registers a callable to convert a bytestring from the database into a custom Python type. The callable will be invoked for all database values that are of the type *typename*. Confer the parameter *detect_types* of the :func:`connect` - function for how the type detection works. Note that the case of *typename* and - the name of the type in your query must match! + function for how the type detection works. Note that *typename* and the name of + the type in your query are matched in case-insensitive manner. .. function:: register_adapter(type, callable) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 89b9ff3..8e90492 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -24,6 +24,9 @@ sockets, both client-side and server-side. This module uses the OpenSSL library. It is available on all modern Unix systems, Windows, Mac OS X, and probably additional platforms, as long as OpenSSL is installed on that platform. +.. versionchanged:: 2.7.13 + Updated to support linking with OpenSSL 1.1.0 + .. note:: Some behavior may be platform dependent, since calls are made to the @@ -211,7 +214,7 @@ instead. The *ciphers* parameter sets the available ciphers for this SSL object. It should be a string in the `OpenSSL cipher list format - <https://wiki.openssl.org/index.php/Manual:Ciphers(1)#CIPHER_LIST_FORMAT>`_. + <https://www.openssl.org/docs/manmaster/man1/ciphers.html>`_. The parameter ``do_handshake_on_connect`` specifies whether to do the SSL handshake automatically after doing a :meth:`socket.connect`, or whether the @@ -291,11 +294,6 @@ purposes. 3DES was dropped from the default cipher string. - .. versionchanged:: 2.7.15 - - TLS 1.3 cipher suites TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, - and TLS_CHACHA20_POLY1305_SHA256 were added to the default cipher string. - .. function:: _https_verify_certificates(enable=True) Specifies whether or not server certificates are verified when creating @@ -744,6 +742,15 @@ Constants .. versionadded:: 2.7.9 +.. data:: OP_ENABLE_MIDDLEBOX_COMPAT + + Send dummy Change Cipher Spec (CCS) messages in TLS 1.3 handshake to make + a TLS 1.3 connection look more like a TLS 1.2 connection. + + This option is only available with OpenSSL 1.1.1 and later. + + .. versionadded:: 2.7.16 + .. data:: OP_NO_COMPRESSION Disable compression on the SSL channel. This is useful if the application @@ -1060,6 +1067,17 @@ to speed up repeated connections from the same clients. :func:`create_default_context` lets the :mod:`ssl` module choose security settings for a given purpose. + .. versionchanged:: 2.7.16 + + The context is created with secure default values. The options + :data:`OP_NO_COMPRESSION`, :data:`OP_CIPHER_SERVER_PREFERENCE`, + :data:`OP_SINGLE_DH_USE`, :data:`OP_SINGLE_ECDH_USE`, + :data:`OP_NO_SSLv2` (except for :data:`PROTOCOL_SSLv2`), + and :data:`OP_NO_SSLv3` (except for :data:`PROTOCOL_SSLv3`) are + set by default. The initial cipher suite list contains only ``HIGH`` + ciphers, no ``NULL`` ciphers and no ``MD5`` ciphers (except for + :data:`PROTOCOL_SSLv2`). + :class:`SSLContext` objects have the following methods and attributes: @@ -1167,7 +1185,7 @@ to speed up repeated connections from the same clients. Set the available ciphers for sockets created with this context. It should be a string in the `OpenSSL cipher list format - <https://wiki.openssl.org/index.php/Manual:Ciphers(1)#CIPHER_LIST_FORMAT>`_. + <https://www.openssl.org/docs/manmaster/man1/ciphers.html>`_. If no cipher can be selected (because compile-time options or other configuration forbids use of all the specified ciphers), an :class:`SSLError` will be raised. @@ -1176,6 +1194,9 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. + OpenSSL 1.1.1 has TLS 1.3 cipher suites enabled by default. The suites + cannot be disabled with :meth:`~SSLContext.set_ciphers`. + .. method:: SSLContext.set_alpn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1392,7 +1413,7 @@ message with one of the parts, you can decrypt it with the other part, and A certificate contains information about two principals. It contains the name of a *subject*, and the subject's public key. It also contains a statement by a -second principal, the *issuer*, that the subject is who he claims to be, and +second principal, the *issuer*, that the subject is who they claim to be, and that this is indeed the subject's public key. The issuer's statement is signed with the issuer's private key, which only the issuer knows. However, anyone can verify the issuer's statement by finding the issuer's public key, decrypting the @@ -1586,7 +1607,7 @@ Visual inspection shows that the certificate does identify the desired service (('commonName', 'www.python.org'),)), 'subjectAltName': (('DNS', 'www.python.org'), ('DNS', 'python.org'), - ('DNS', 'pypi.python.org'), + ('DNS', 'pypi.org'), ('DNS', 'docs.python.org'), ('DNS', 'testpypi.python.org'), ('DNS', 'bugs.python.org'), diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index e16bd77..ff68738 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1381,7 +1381,7 @@ string functions based on regular expressions. .. method:: str.upper() Return a copy of the string with all the cased characters [4]_ converted to - uppercase. Note that ``str.upper().isupper()`` might be ``False`` if ``s`` + uppercase. Note that ``s.upper().isupper()`` might be ``False`` if ``s`` contains uncased characters or if the Unicode category of the resulting character(s) is not "Lu" (Letter, uppercase), but e.g. "Lt" (Letter, titlecase). diff --git a/Doc/library/string.rst b/Doc/library/string.rst index c2af446..7eedc86 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -257,8 +257,9 @@ attribute using :func:`getattr`, while an expression of the form ``'[index]'`` does an index lookup using :func:`__getitem__`. .. versionchanged:: 2.7 - The positional argument specifiers can be omitted, so ``'{} {}'`` is - equivalent to ``'{0} {1}'``. + The positional argument specifiers can be omitted for :meth:`str.format` and + :meth:`unicode.format`, so ``'{} {}'`` is equivalent to ``'{0} {1}'``, + ``u'{} {}'`` is equivalent to ``u'{0} {1}'``. Some simple format string examples:: @@ -463,10 +464,10 @@ The available presentation types for floating point and decimal values are: | ``'E'`` | Exponent notation. Same as ``'e'`` except it uses an | | | upper case 'E' as the separator character. | +---------+----------------------------------------------------------+ - | ``'f'`` | Fixed point. Displays the number as a fixed-point | - | | number. The default precision is ``6``. | + | ``'f'`` | Fixed-point notation. Displays the number as a | + | | fixed-point number. The default precision is ``6``. | +---------+----------------------------------------------------------+ - | ``'F'`` | Fixed point. Same as ``'f'``. | + | ``'F'`` | Fixed point notation. Same as ``'f'``. | +---------+----------------------------------------------------------+ | ``'g'`` | General format. For a given precision ``p >= 1``, | | | this rounds the number to ``p`` significant digits and | @@ -521,7 +522,7 @@ addition of the ``{}`` and with ``:`` used instead of ``%``. For example, ``'%03.2f'`` can be translated to ``'{:03.2f}'``. The new format syntax also supports new and different options, shown in the -follow examples. +following examples. Accessing arguments by position:: @@ -700,7 +701,7 @@ these rules. The methods of :class:`Template` are: simply return ``$`` instead of raising :exc:`ValueError`. While other exceptions may still occur, this method is called "safe" - because substitutions always tries to return a usable string instead of + because it always tries to return a usable string instead of raising an exception. In another sense, :meth:`safe_substitute` may be anything other than safe, since it will silently ignore malformed templates containing dangling delimiters, unmatched braces, or diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 284cf44..53a0161 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -32,7 +32,7 @@ functions can be found in the subprocess-replacements_ section. :pep:`324` -- PEP proposing the subprocess module -.. _subprocess32: https://pypi.python.org/pypi/subprocess32/ +.. _subprocess32: https://pypi.org/project/subprocess32/ Using the :mod:`subprocess` Module ---------------------------------- diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index e8fc09e..7655c60 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -83,7 +83,7 @@ Python currently supports seven schemes: - *nt*: scheme for NT platforms like Windows. - *nt_user*: scheme for NT platforms, when the *user* option is used. - *os2*: scheme for OS/2 platforms. -- *os2_home*: scheme for OS/2 patforms, when the *user* option is used. +- *os2_home*: scheme for OS/2 platforms, when the *user* option is used. Each scheme is itself composed of a series of paths and each path has a unique identifier. Python currently uses eight paths: @@ -188,7 +188,7 @@ Other functions Windows will return one of: - - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc) + - win-amd64 (64bit Windows on AMD64, aka x86_64, Intel64, and EM64T) - win-ia64 (64bit Windows on Itanium) - win32 (all others - specifically, sys.platform is returned) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index c819bf5..5789e5e 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -37,6 +37,10 @@ Some facts and figures: character devices and block devices and is able to acquire and restore file information like timestamp, access permissions and owner. +.. note:: + Handling of multi-stream bzip2 files is not supported. Modules such as + `bz2file <https://github.com/nvawda/bz2file>`_ let you overcome this. + .. function:: open(name=None, mode='r', fileobj=None, bufsize=10240, \*\*kwargs) diff --git a/Doc/library/time.rst b/Doc/library/time.rst index b2a9253..48a01db 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -236,7 +236,7 @@ The module defines the following functions and data items: argument. If *t* is not provided, the current time as returned by :func:`localtime` is used. *format* must be a string. :exc:`ValueError` is raised if any field in *t* is outside of the allowed range. :func:`strftime` - returns a locale depedent byte string; the result may be converted to unicode + returns a locale dependent byte string; the result may be converted to unicode by doing ``strftime(<myformat>).decode(locale.getlocale()[1])``. .. versionchanged:: 2.1 diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index ce5e63a..3356e4e 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -11,6 +11,11 @@ the Tk GUI toolkit. Both Tk and :mod:`Tkinter` are available on most Unix platforms, as well as on Windows systems. (Tk itself is not part of Python; it is maintained at ActiveState.) +Running ``python -m Tkinter`` from the command line should open a window +demonstrating a simple Tk interface, letting you know that :mod:`Tkinter` is +properly installed on your system, and also showing what version of Tcl/Tk is +installed, so you can read the Tcl/Tk documentation specific to that version. + .. note:: :mod:`Tkinter` has been renamed to :mod:`tkinter` in Python 3. The @@ -19,6 +24,8 @@ is maintained at ActiveState.) .. seealso:: + Tkinter documentation: + `Python Tkinter Resources <https://wiki.python.org/moin/TkInter>`_ The Python Tkinter Topic Guide provides a great deal of information on using Tk from Python and links to other sources of information on Tk. @@ -32,17 +39,32 @@ is maintained at ActiveState.) `Tkinter docs from effbot <http://effbot.org/tkinterbook/>`_ Online reference for tkinter supported by effbot.org. - `Tcl/Tk manual <https://www.tcl.tk/man/tcl8.5/>`_ - Official manual for the latest tcl/tk version. - - `Programming Python <http://learning-python.com/books/about-pp4e.html>`_ + `Programming Python <http://learning-python.com/about-pp4e.html>`_ Book by Mark Lutz, has excellent coverage of Tkinter. - `Modern Tkinter for Busy Python Developers <http://www.amazon.com/Modern-Tkinter-Python-Developers-ebook/dp/B0071QDNLO/>`_ + `Modern Tkinter for Busy Python Developers <https://www.amazon.com/Modern-Tkinter-Python-Developers-ebook/dp/B0071QDNLO/>`_ Book by Mark Rozerman about building attractive and modern graphical user interfaces with Python and Tkinter. `Python and Tkinter Programming <https://www.manning.com/books/python-and-tkinter-programming>`_ - The book by John Grayson (ISBN 1-884777-81-3). + Book by John Grayson (ISBN 1-884777-81-3). + + Tcl/Tk documentation: + + `Tk commands <https://www.tcl.tk/man/tcl8.6/TkCmd/contents.htm>`_ + Most commands are available as :mod:`Tkinter` or :mod:`Tkinter.ttk` classes. + Change '8.6' to match the version of your Tcl/Tk installation. + + `Tcl/Tk recent man pages <https://www.tcl.tk/doc/>`_ + Recent Tcl/Tk manuals on www.tcl.tk. + + `ActiveState Tcl Home Page <http://tcl.activestate.com/>`_ + The Tk/Tcl development is largely taking place at ActiveState. + + `Tcl and the Tk Toolkit <https://www.amazon.com/exec/obidos/ASIN/020163337X>`_ + Book by John Ousterhout, the inventor of Tcl. + + `Practical Programming in Tcl and Tk <http://www.beedub.com/book/>`_ + Brent Welch's encyclopedic book. Tkinter Modules @@ -182,18 +204,6 @@ documentation that exists. Here are some hints: when nothing else makes sense. -.. seealso:: - - `ActiveState Tcl Home Page <http://tcl.activestate.com/>`_ - The Tk/Tcl development is largely taking place at ActiveState. - - `Tcl and the Tk Toolkit <http://www.amazon.com/exec/obidos/ASIN/020163337X>`_ - The book by John Ousterhout, the inventor of Tcl. - - `Practical Programming in Tcl and Tk <http://www.beedub.com/book/>`_ - Brent Welch's encyclopedic book. - - A Simple Hello World Program ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -801,12 +811,13 @@ Menu indexes (menu.invoke(), menu.entryconfig(), etc.) Images ^^^^^^ -Bitmap/Pixelmap images can be created through the subclasses of -:class:`Tkinter.Image`: +Images of different formats can be created through the corresponding subclass +of :class:`Tkinter.Image`: -* :class:`BitmapImage` can be used for X11 bitmap data. +* :class:`BitmapImage` for images in XBM format. -* :class:`PhotoImage` can be used for GIF and PPM/PGM color bitmaps. +* :class:`PhotoImage` for images in PGM, PPM, GIF and PNG formats. The latter + is supported starting with Tk 8.6. Either type of image is created through either the ``file`` or the ``data`` option (other options are available as well). @@ -817,6 +828,10 @@ reference to the image. When the last Python reference to the image object is deleted, the image data is deleted as well, and Tk will display an empty box wherever the image was used. +.. seealso:: + + The `Pillow <http://python-pillow.org/>`_ package adds support for + formats such as BMP, JPEG, TIFF, and WebP, among others. .. _tkinter-file-handlers: diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 1c8a967..63c0bee 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1229,7 +1229,7 @@ Using events :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas - :param num: number of the mouse-button, defaults to 1 (left mouse button) + :param btn: number of the mouse-button, defaults to 1 (left mouse button) :param add: ``True`` or ``False`` -- if ``True``, a new binding will be added, otherwise it will replace a former binding @@ -1250,7 +1250,7 @@ Using events :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas - :param num: number of the mouse-button, defaults to 1 (left mouse button) + :param btn: number of the mouse-button, defaults to 1 (left mouse button) :param add: ``True`` or ``False`` -- if ``True``, a new binding will be added, otherwise it will replace a former binding @@ -1274,7 +1274,7 @@ Using events :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas - :param num: number of the mouse-button, defaults to 1 (left mouse button) + :param btn: number of the mouse-button, defaults to 1 (left mouse button) :param add: ``True`` or ``False`` -- if ``True``, a new binding will be added, otherwise it will replace a former binding @@ -1656,7 +1656,7 @@ Using screen events :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas - :param num: number of the mouse-button, defaults to 1 (left mouse button) + :param btn: number of the mouse-button, defaults to 1 (left mouse button) :param add: ``True`` or ``False`` -- if ``True``, a new binding will be added, otherwise it will replace a former binding diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index b36126e..2638d6c 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -78,7 +78,7 @@ need to derive from a specific class. Module :mod:`doctest` Another test-support module with a very different flavor. - `unittest2: A backport of new unittest features for Python 2.4-2.6 <https://pypi.python.org/pypi/unittest2>`_ + `unittest2: A backport of new unittest features for Python 2.4-2.6 <https://pypi.org/project/unittest2>`_ Many new features were added to unittest in Python 2.7, including test discovery. unittest2 allows you to use these features with earlier versions of Python. @@ -87,7 +87,7 @@ need to derive from a specific class. Kent Beck's original paper on testing frameworks using the pattern shared by :mod:`unittest`. - `Nose <https://nose.readthedocs.org/en/latest/>`_ and `py.test <http://pytest.org>`_ + `Nose <https://nose.readthedocs.io/>`_ and `pytest <https://docs.pytest.org/>`_ Third-party unittest frameworks with a lighter-weight syntax for writing tests. For example, ``assert func(10) == 42``. @@ -730,7 +730,7 @@ Test cases .. method:: setUpClass() - A class method called before tests in an individual class run. + A class method called before tests in an individual class are run. ``setUpClass`` is called with the class as the only argument and must be decorated as a :func:`classmethod`:: diff --git a/Doc/library/urllib.rst b/Doc/library/urllib.rst index 2a5ea71..084d567 100644 --- a/Doc/library/urllib.rst +++ b/Doc/library/urllib.rst @@ -147,14 +147,15 @@ High-level interface :envvar:`no_proxy` environment variable. .. versionchanged:: 2.7.9 - The *context* parameter was added. All the neccessary certificate and hostname checks are done by default. + The *context* parameter was added. All the neccessary certificate and hostname + checks are done by default. .. deprecated:: 2.6 The :func:`urlopen` function has been removed in Python 3 in favor of :func:`urllib2.urlopen`. -.. function:: urlretrieve(url[, filename[, reporthook[, data]]]) +.. function:: urlretrieve(url[, filename[, reporthook[, data[, context]]]]) Copy a network object denoted by a URL to a local file, if necessary. If the URL points to a local file, or a valid cached copy of the object exists, the object @@ -166,9 +167,9 @@ High-level interface The second argument, if present, specifies the file location to copy to (if absent, the location will be a tempfile with a generated name). The third - argument, if present, is a hook function that will be called once on + argument, if present, is a callable that will be called once on establishment of the network connection and once after each block read - thereafter. The hook will be passed three arguments; a count of blocks + thereafter. The callable will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. The third argument may be ``-1`` on older FTP servers which do not return a file size in response to a retrieval request. @@ -179,6 +180,10 @@ High-level interface :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urlencode` function below. + The *context* parameter may be set to a :class:`ssl.SSLContext` instance to + configure the SSL settings that are used if :func:`urlretrieve` makes a HTTPS + connection. + .. versionchanged:: 2.5 :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that the amount of data available was less than the expected amount (which is the @@ -196,6 +201,10 @@ High-level interface the size of the data it has downloaded, and just returns it. In this case you just have to assume that the download was successful. + .. versionchanged:: 2.7.9 + The *context* parameter was added. All the neccessary certificate and hostname + checks are done by default. + .. data:: _urlopener @@ -349,6 +358,10 @@ URL Opener objects :class:`URLopener` objects will raise an :exc:`IOError` exception if the server returns an error code. + .. versionchanged:: 2.7.9 + The *context* parameter was added. All the neccessary certificate and hostname + checks are done by default. + .. method:: open(fullurl[, data]) Open *fullurl* using the appropriate protocol. This method sets up cache and diff --git a/Doc/library/urlparse.rst b/Doc/library/urlparse.rst index b933dda..22249da 100644 --- a/Doc/library/urlparse.rst +++ b/Doc/library/urlparse.rst @@ -126,7 +126,7 @@ The :mod:`urlparse` module defines the following functions: Added IPv6 URL parsing capabilities. -.. function:: parse_qs(qs[, keep_blank_values[, strict_parsing]]) +.. function:: parse_qs(qs[, keep_blank_values[, strict_parsing[, max_num_fields]]]) Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a @@ -143,14 +143,20 @@ The :mod:`urlparse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional argument *max_num_fields* is the maximum number of fields to + read. If set, then throws a :exc:`ValueError` if there are more than + *max_num_fields* fields read. + Use the :func:`urllib.urlencode` function to convert such dictionaries into query strings. .. versionadded:: 2.6 Copied from the :mod:`cgi` module. + .. versionchanged:: 2.7.16 + Added *max_num_fields* parameter. -.. function:: parse_qsl(qs[, keep_blank_values[, strict_parsing]]) +.. function:: parse_qsl(qs[, keep_blank_values[, strict_parsing[, max_num_fields]]]) Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of @@ -166,12 +172,18 @@ The :mod:`urlparse` module defines the following functions: parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. + The optional argument *max_num_fields* is the maximum number of fields to + read. If set, then throws a :exc:`ValueError` if there are more than + *max_num_fields* fields read. + Use the :func:`urllib.urlencode` function to convert such lists of pairs into query strings. .. versionadded:: 2.6 Copied from the :mod:`cgi` module. + .. versionchanged:: 2.7.16 + Added *max_num_fields* parameter. .. function:: urlunparse(parts) diff --git a/Doc/library/uuid.rst b/Doc/library/uuid.rst index bc69b5f..d2808cd 100644 --- a/Doc/library/uuid.rst +++ b/Doc/library/uuid.rst @@ -23,12 +23,13 @@ random UUID. .. class:: UUID([hex[, bytes[, bytes_le[, fields[, int[, version]]]]]]) Create a UUID from either a string of 32 hexadecimal digits, a string of 16 - bytes as the *bytes* argument, a string of 16 bytes in little-endian order as - the *bytes_le* argument, a tuple of six integers (32-bit *time_low*, 16-bit - *time_mid*, 16-bit *time_hi_version*, 8-bit *clock_seq_hi_variant*, 8-bit - *clock_seq_low*, 48-bit *node*) as the *fields* argument, or a single 128-bit - integer as the *int* argument. When a string of hex digits is given, curly - braces, hyphens, and a URN prefix are all optional. For example, these + bytes in big-endian order as the *bytes* argument, a string of 16 bytes in + little-endian order as the *bytes_le* argument, a tuple of six integers + (32-bit *time_low*, 16-bit *time_mid*, 16-bit *time_hi_version*, + 8-bit *clock_seq_hi_variant*, 8-bit *clock_seq_low*, 48-bit *node*) as the + *fields* argument, or a single 128-bit integer as the *int* argument. + When a string of hex digits is given, curly braces, hyphens, + and a URN prefix are all optional. For example, these expressions all yield the same UUID:: UUID('{12345678-1234-5678-1234-567812345678}') diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index f91259a..2df5bdd 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -169,7 +169,7 @@ module documentation. This section lists the differences between the API and the *encoding* argument was introduced; see :meth:`writexml`. -.. method:: Node.toprettyxml([indent=""[, newl=""[, encoding=""]]]) +.. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None) Return a pretty-printed version of the document. *indent* specifies the indentation string and defaults to a tabulator; *newl* specifies the string diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst index a8f20cd..b107c29 100644 --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -129,8 +129,8 @@ break backward compatibility. After all inline DTD and entity expansion are well-defined XML features. -.. _defusedxml: https://pypi.python.org/pypi/defusedxml/ -.. _defusedexpat: https://pypi.python.org/pypi/defusedexpat/ +.. _defusedxml: https://pypi.org/project/defusedxml/ +.. _defusedexpat: https://pypi.org/project/defusedexpat/ .. _Billion Laughs: https://en.wikipedia.org/wiki/Billion_laughs .. _ZIP bomb: https://en.wikipedia.org/wiki/Zip_bomb .. _DTD: https://en.wikipedia.org/wiki/Document_type_definition diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index 25e4fa9..9ec9696 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -31,7 +31,7 @@ The convenience functions are: Create and return a SAX :class:`~xml.sax.xmlreader.XMLReader` object. The first parser found will - be used. If *parser_list* is provided, it must be a sequence of strings which + be used. If *parser_list* is provided, it must be a list of strings which name modules that have a function named :func:`create_parser`. Modules listed in *parser_list* will be used before modules in the default list of parsers. diff --git a/Doc/license.rst b/Doc/license.rst index f33495a..eded4a9 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -87,7 +87,7 @@ PSF LICENSE AGREEMENT FOR PYTHON |release| analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python |release| alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright © 2001-2018 Python Software Foundation; All Rights + copyright, i.e., "Copyright © 2001-2019 Python Software Foundation; All Rights Reserved" are retained in Python |release| alone or in any derivative version prepared by Licensee. diff --git a/Doc/make.bat b/Doc/make.bat index d8e8df0..2e199f1 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -5,23 +5,37 @@ pushd %~dp0 set this=%~n0
-call ..\PCBuild\find_python.bat %PYTHON%
-if not defined SPHINXBUILD if defined PYTHON (
+call ..\PCbuild\find_python.bat %PYTHON%
+
+if not defined PYTHON set PYTHON=py
+
+if not defined SPHINXBUILD (
%PYTHON% -c "import sphinx" > nul 2> nul
if errorlevel 1 (
echo Installing sphinx with %PYTHON%
%PYTHON% -m pip install sphinx
if errorlevel 1 exit /B
)
- set SPHINXBUILD=%PYTHON% -c "import sphinx, sys; sys.argv[0] = 'sphinx-build'; sphinx.main()"
+ set SPHINXBUILD=%PYTHON% -c "import sphinx, sys; sys.argv[0] = 'sphinx-build'; sys.exit(sphinx.main())"
)
-if not defined PYTHON set PYTHON=py
-if not defined SPHINXBUILD set SPHINXBUILD=sphinx-build
+if "%1" NEQ "htmlhelp" goto :skiphhcsearch
+if exist "%HTMLHELP%" goto :skiphhcsearch
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+rem Search for HHC in likely places
+set HTMLHELP=
+where hhc /q && set HTMLHELP=hhc && goto :skiphhcsearch
+where /R ..\externals hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles(x86)%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" (
+ echo.
+ echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
+ echo.to the path to hhc.exe or download and install it from
+ echo.http://msdn.microsoft.com/en-us/library/ms669985
+ exit /B 1
+)
+:skiphhcsearch
if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
@@ -33,11 +47,11 @@ if "%1" EQU "help" goto help if "%1" EQU "check" goto check
if "%1" EQU "serve" goto serve
if "%1" == "clean" (
- rmdir /q /s %BUILDDIR%
+ rmdir /q /s "%BUILDDIR%"
goto end
)
-%SPHINXBUILD% 2> nul
+%SPHINXBUILD% >nul 2> nul
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
@@ -47,7 +61,8 @@ if errorlevel 9009 ( echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
- goto end
+ popd
+ exit /B 1
)
rem Targets that do require sphinx-build and have their own label
@@ -81,22 +96,14 @@ echo.be passed by setting the SPHINXOPTS environment variable. goto end
:build
+if not exist "%BUILDDIR%" mkdir "%BUILDDIR%"
if NOT "%PAPER%" == "" (
set SPHINXOPTS=-D latex_elements.papersize=%PAPER% %SPHINXOPTS%
)
-cmd /C "%SPHINXBUILD% %SPHINXOPTS% -b%1 -dbuild\doctrees . %BUILDDIR%\%*"
+cmd /S /C "%SPHINXBUILD% %SPHINXOPTS% -b%1 -dbuild\doctrees . "%BUILDDIR%\%1" %2 %3 %4 %5 %6 %7 %8 %9"
if "%1" EQU "htmlhelp" (
- if not exist "%HTMLHELP%" (
- echo.
- echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
- echo.to the path to hhc.exe or download and install it from
- echo.http://msdn.microsoft.com/en-us/library/ms669985
- rem Set errorlevel to 1 and exit
- cmd /C exit /b 1
- goto end
- )
- cmd /C "%HTMLHELP%" build\htmlhelp\python%DISTVERSION:.=%.hhp
+ "%HTMLHELP%" "%BUILDDIR%\htmlhelp\python%DISTVERSION:.=%.hhp"
rem hhc.exe seems to always exit with code 1, reset to 0 for less than 2
if not errorlevel 2 cmd /C exit /b 0
)
@@ -116,19 +123,19 @@ if NOT "%2" EQU "" ( )
cmd /C %this% html
-if EXIST %BUILDDIR%\html\index.html (
- echo.Opening %BUILDDIR%\html\index.html in the default web browser...
- start %BUILDDIR%\html\index.html
+if EXIST "%BUILDDIR%\html\index.html" (
+ echo.Opening "%BUILDDIR%\html\index.html" in the default web browser...
+ start "" "%BUILDDIR%\html\index.html"
)
goto end
:check
-cmd /C %PYTHON% tools\rstlint.py -i tools
+cmd /S /C "%PYTHON% tools\rstlint.py -i tools"
goto end
:serve
-cmd /C %PYTHON% ..\Tools\scripts\serve.py %BUILDDIR%\html
+cmd /S /C "%PYTHON% ..\Tools\scripts\serve.py "%BUILDDIR%\html""
goto end
:end
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 66ae530..523d9b5 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -185,7 +185,7 @@ effect of Pascal's ``for i := a to b do``; e.g., ``range(3)`` returns the list single: mutable sequence; loop over There is a subtlety when the sequence is being modified by the loop (this can - only occur for mutable sequences, i.e. lists). An internal counter is used to + only occur for mutable sequences, e.g. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) @@ -281,9 +281,11 @@ function that handled an exception. statement: break statement: continue -The optional :keyword:`else` clause is executed if and when control flows off -the end of the :keyword:`try` clause. [#]_ Exceptions in the :keyword:`else` -clause are not handled by the preceding :keyword:`except` clauses. +The optional :keyword:`else` clause is executed if the control flow leaves the +:keyword:`try` suite, no exception was raised, and no :keyword:`return`, +:keyword:`continue`, or :keyword:`break` statement was executed. Exceptions in +the :keyword:`else` clause are not handled by the preceding :keyword:`except` +clauses. .. index:: keyword: finally @@ -596,10 +598,6 @@ which is then bound to the class name. there is a :keyword:`finally` clause which happens to raise another exception. That new exception causes the old one to be lost. -.. [#] Currently, control "flows off the end" except in the case of an exception or the - execution of a :keyword:`return`, :keyword:`continue`, or :keyword:`break` - statement. - .. [#] A string literal appearing as the first statement in the function body is transformed into the function's ``__doc__`` attribute and therefore the function's :term:`docstring`. diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 7bdb141..92f95c2 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1919,12 +1919,6 @@ sequences, it should iterate through the values. indexes to allow proper detection of the end of the sequence. -.. method:: object.__missing__(self, key) - - Called by :class:`dict`\ .\ :meth:`__getitem__` to implement ``self[key]`` for dict subclasses - when key is not in the dictionary. - - .. method:: object.__setitem__(self, key, value) Called to implement assignment to ``self[key]``. Same note as for @@ -1943,6 +1937,12 @@ sequences, it should iterate through the values. values as for the :meth:`__getitem__` method. +.. method:: object.__missing__(self, key) + + Called by :class:`dict`\ .\ :meth:`__getitem__` to implement ``self[key]`` for dict subclasses + when key is not in the dictionary. + + .. method:: object.__iter__(self) This method is called when an iterator is required for a container. This method diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 8556fa8..6afc867 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -581,7 +581,7 @@ whose value is one of the keys of the mapping, and the subscription selects the value in the mapping that corresponds to that key. (The expression list is a tuple except if it has exactly one item.) -If the primary is a sequence, the expression (list) must evaluate to a plain +If the primary is a sequence, the expression list must evaluate to a plain integer. If this value is negative, the length of the sequence is added to it (so that, e.g., ``x[-1]`` selects the last item of ``x``.) The resulting value must be a nonnegative integer less than the number of items in the sequence, and @@ -1390,10 +1390,10 @@ Lambdas Lambda expressions (sometimes called lambda forms) have the same syntactic position as expressions. They are a shorthand to create anonymous functions; the expression -``lambda arguments: expression`` yields a function object. The unnamed object +``lambda parameters: expression`` yields a function object. The unnamed object behaves like a function object defined with :: - def name(arguments): + def <lambda>(parameters): return expression See section :ref:`function` for the syntax of parameter lists. Note that diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 9f69ef7..435dfc5 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -73,11 +73,12 @@ Physical lines -------------- A physical line is a sequence of characters terminated by an end-of-line -sequence. In source files, any of the standard platform line termination -sequences can be used - the Unix form using ASCII LF (linefeed), the Windows -form using the ASCII sequence CR LF (return followed by linefeed), or the old -Macintosh form using the ASCII CR (return) character. All of these forms can be -used equally, regardless of platform. +sequence. In source files and strings, any of the standard platform line +termination sequences can be used - the Unix form using ASCII LF (linefeed), +the Windows form using the ASCII sequence CR LF (return followed by linefeed), +or the old Macintosh form using the ASCII CR (return) character. All of these +forms can be used equally, regardless of platform. The end of input also serves +as an implicit terminator for the final physical line. When embedding Python, source code strings should be passed to Python APIs using the standard C conventions for newline characters (the ``\n`` character, diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 43935da..e2d643f 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -25,6 +25,7 @@ simple statements is: : | `break_stmt` : | `continue_stmt` : | `import_stmt` + : | `future_stmt` : | `global_stmt` : | `exec_stmt` diff --git a/Doc/reference/toplevel_components.rst b/Doc/reference/toplevel_components.rst index 304abac..44196f7 100644 --- a/Doc/reference/toplevel_components.rst +++ b/Doc/reference/toplevel_components.rst @@ -51,11 +51,11 @@ a complete program; each statement is executed in the namespace of single: command line single: standard input -Under Unix, a complete program can be passed to the interpreter in three forms: -with the :option:`-c` *string* command line option, as a file passed as the -first command line argument, or as standard input. If the file or standard input -is a tty device, the interpreter enters interactive mode; otherwise, it executes -the file as a complete program. +A complete program can be passed to the interpreter +in three forms: with the :option:`-c` *string* command line option, as a file +passed as the first command line argument, or as standard input. If the file +or standard input is a tty device, the interpreter enters interactive mode; +otherwise, it executes the file as a complete program. .. _file-input: diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 7bbd580..6378f76 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -15,9 +15,11 @@ SOURCE_URI = 'https://github.com/python/cpython/tree/2.7/%s' from docutils import nodes, utils from docutils.parsers.rst import Directive +from sphinx.util import status_iterator from sphinx.util.nodes import split_explicit_title from sphinx.writers.html import HTMLTranslator from sphinx.writers.latex import LaTeXTranslator +from sphinx.writers.text import TextTranslator # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body @@ -158,8 +160,11 @@ from sphinx.writers.text import TextWriter class PydocTopicsBuilder(Builder): name = 'pydoc-topics' + default_translator_class = TextTranslator + def init(self): self.topics = {} + self.secnumbers = {} def get_outdated_docs(self): return 'all pydoc topics' @@ -169,9 +174,9 @@ class PydocTopicsBuilder(Builder): def write(self, *ignored): writer = TextWriter(self) - for label in self.status_iterator(pydoc_topic_labels, - 'building topics... ', - length=len(pydoc_topic_labels)): + for label in status_iterator(pydoc_topic_labels, + 'building topics... ', + length=len(pydoc_topic_labels)): if label not in self.env.domaindata['std']['labels']: self.warn('label %r not in documentation' % label) continue @@ -248,11 +253,9 @@ def setup(app): app.add_directive('impl-detail', ImplementationDetail) app.add_builder(PydocTopicsBuilder) app.add_builder(suspicious.CheckSuspiciousMarkupBuilder) - app.add_description_unit('opcode', 'opcode', '%s (opcode)', - parse_opcode_signature) - app.add_description_unit('pdbcommand', 'pdbcmd', '%s (pdb command)', - parse_pdb_command) - app.add_description_unit('2to3fixer', '2to3fixer', '%s (2to3 fixer)') + app.add_object_type('opcode', 'opcode', '%s (opcode)', parse_opcode_signature) + app.add_object_type('pdbcommand', 'pdbcmd', '%s (pdb command)', parse_pdb_command) + app.add_object_type('2to3fixer', '2to3fixer', '%s (2to3 fixer)') app.add_directive_to_domain('py', 'decorator', PyDecoratorFunction) app.add_directive_to_domain('py', 'decoratormethod', PyDecoratorMethod) return {'version': '1.0', 'parallel_read_safe': True} diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js index 8e0c5ea..dbf907e 100644 --- a/Doc/tools/static/switchers.js +++ b/Doc/tools/static/switchers.js @@ -11,7 +11,7 @@ var all_versions = { '3.8': 'dev (3.8)', - '3.7': 'pre (3.7)', + '3.7': '3.7', '3.6': '3.6', '3.5': '3.5', '2.7': '2.7', @@ -48,6 +48,12 @@ else buf.push('<option value="' + language + '">' + title + '</option>'); }); + if (!(current_language in all_languages)) { + // In case we're browsing a language that is not yet in all_languages. + buf.push('<option value="' + current_language + '" selected="selected">' + + current_language + '</option>'); + all_languages[current_language] = current_language; + } buf.push('</select>'); return buf.join(''); } diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index ddd8d75..1181f71 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -1,12 +1,13 @@ <h3>{% trans %}Download{% endtrans %}</h3> <p><a href="{{ pathto('download') }}">{% trans %}Download these documents{% endtrans %}</a></p> -<h3>{% trans %}Docs for other versions{% endtrans %}</h3> +<h3>{% trans %}Docs by version{% endtrans %}</h3> <ul> <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (in development){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (pre-release){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (stable){% endtrans %}</a></li> - <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (stable){% endtrans %}</a></li> - <li><a href="https://www.python.org/doc/versions/">{% trans %}Old versions{% endtrans %}</a></li> + <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li> + <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li> + <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li> + <li><a href="https://docs.python.org/2.7/">{% trans %}Python 2.7 (stable){% endtrans %}</a></li> + <li><a href="https://www.python.org/doc/versions/">{% trans %}All versions{% endtrans %}</a></li> </ul> <h3>{% trans %}Other resources{% endtrans %}</h3> diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 7f45c76..b17e150 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -328,8 +328,8 @@ the corresponding function with an argument list that is created by inserting the method's object before the first argument. If you still don't understand how methods work, a look at the implementation can -perhaps clarify matters. When an instance attribute is referenced that isn't a -data attribute, its class is searched. If the name denotes a valid class +perhaps clarify matters. When a non-data attribute of an instance is +referenced, the instance's class is searched. If the name denotes a valid class attribute that is a function object, a method object is created by packing (pointers to) the instance object and the function object just found together in an abstract object: this is the method object. When the method object is called @@ -620,6 +620,9 @@ be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice. +.. index:: + pair: name; mangling + Since there is a valid use-case for class-private members (namely to avoid name clashes of names with names defined by subclasses), there is limited support for such a mechanism, called :dfn:`name mangling`. Any identifier of the form @@ -651,6 +654,11 @@ breaking intraclass method calls. For example:: for item in zip(keys, values): self.items_list.append(item) +The above example would work even if ``MappingSubclass`` were to introduce a +``__update`` identifier since it is replaced with ``_Mapping__update`` in the +``Mapping`` class and ``_MappingSubclass__update`` in the ``MappingSubclass`` +class respectively. + Note that the mangling rules are designed mostly to avoid accidents; it still is possible to access or modify a variable that is considered private. This can even be useful in special circumstances, such as in the debugger. diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 466009e..247dda7 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -315,7 +315,7 @@ to create specific exception classes for different error conditions:: self.next = next self.msg = msg -Most exceptions are defined with names that end in "Error," similar to the +Most exceptions are defined with names that end in "Error", similar to the naming of the standard exceptions. Many standard modules define their own exceptions to report errors that may diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index 6c8609f..b727be0 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -137,12 +137,12 @@ where *encoding* is one of the valid :mod:`codecs` supported by Python. For example, to declare that Windows-1252 encoding is to be used, the first line of your source code file should be:: - # -*- coding: cp-1252 -*- + # -*- coding: cp1252 -*- One exception to the *first line* rule is when the source code starts with a :ref:`UNIX "shebang" line <tut-scripts>`. In this case, the encoding declaration should be added as the second line of the file. For example:: #!/usr/bin/env python - # -*- coding: cp-1252 -*- + # -*- coding: cp1252 -*- diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index be9aa83..2ff776e 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -149,12 +149,12 @@ to escape quotes:: "doesn't" >>> "doesn't" # ...or use double quotes instead "doesn't" - >>> '"Yes," he said.' - '"Yes," he said.' - >>> "\"Yes,\" he said." - '"Yes," he said.' - >>> '"Isn\'t," she said.' - '"Isn\'t," she said.' + >>> '"Yes," they said.' + '"Yes," they said.' + >>> "\"Yes,\" they said." + '"Yes," they said.' + >>> '"Isn\'t," they said.' + '"Isn\'t," they said.' In the interactive interpreter, the output string is enclosed in quotes and special characters are escaped with backslashes. While this might sometimes @@ -165,10 +165,10 @@ enclosed in single quotes. The :keyword:`print` statement produces a more readable output, by omitting the enclosing quotes and by printing escaped and special characters:: - >>> '"Isn\'t," she said.' - '"Isn\'t," she said.' - >>> print '"Isn\'t," she said.' - "Isn't," she said. + >>> '"Isn\'t," they said.' + '"Isn\'t," they said.' + >>> print '"Isn\'t," they said.' + "Isn't," they said. >>> s = 'First line.\nSecond line.' # \n means newline >>> s # without print, \n is included in the output 'First line.\nSecond line.' diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index ec3bd9c..042d233 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -377,7 +377,7 @@ module names". For example, the module name :mod:`A.B` designates a submodule named ``B`` in a package named ``A``. Just like the use of modules saves the authors of different modules from having to worry about each other's global variable names, the use of dotted module names saves the authors of multi-module -packages like NumPy or the Python Imaging Library from having to worry about +packages like NumPy or Pillow from having to worry about each other's module names. Suppose you want to design a collection of modules (a "package") for the uniform diff --git a/Doc/tutorial/whatnow.rst b/Doc/tutorial/whatnow.rst index 89dfa6a..f6ce4f4 100644 --- a/Doc/tutorial/whatnow.rst +++ b/Doc/tutorial/whatnow.rst @@ -38,7 +38,7 @@ More Python resources: * https://docs.python.org: Fast access to Python's documentation. -* https://pypi.python.org/pypi: The Python Package Index, previously also nicknamed +* https://pypi.org: The Python Package Index, previously also nicknamed the Cheese Shop, is an index of user-created Python modules that are available for download. Once you begin releasing code, you can register it here so that others can find it. diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst index 7719baa..4789e4f 100644 --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -142,7 +142,7 @@ Editors and IDEs ================ There are a number of IDEs that support Python programming language. -Many editors and IDEs provide syntax highlighting, debugging tools, and PEP-8 checks. +Many editors and IDEs provide syntax highlighting, debugging tools, and :pep:`8` checks. Please go to `Python Editors <https://wiki.python.org/moin/PythonEditors>`_ and `Integrated Development Environments <https://wiki.python.org/moin/IntegratedDevelopmentEnvironments>`_ diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index f597476..778b884 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -242,7 +242,7 @@ The Windows-specific standard modules are documented in PyWin32 ------- -The `PyWin32 <https://pypi.python.org/pypi/pywin32>`_ module by Mark Hammond +The `PyWin32 <https://pypi.org/project/pywin32>`_ module by Mark Hammond is a collection of modules for advanced Windows-specific support. This includes utilities for: diff --git a/Doc/whatsnew/2.3.rst b/Doc/whatsnew/2.3.rst index 838ca38..1ef2cb5 100644 --- a/Doc/whatsnew/2.3.rst +++ b/Doc/whatsnew/2.3.rst @@ -659,7 +659,7 @@ The heart of the catalog is the new Distutils :command:`register` command. Running ``python setup.py register`` will collect the metadata describing a package, such as its name, version, maintainer, description, &c., and send it to a central catalog server. The resulting catalog is available from -https://pypi.python.org/pypi. +https://pypi.org. To make the catalog a bit more useful, a new optional *classifiers* keyword argument has been added to the Distutils :func:`setup` function. A list of diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index d2395e7..6400bb1 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -229,7 +229,7 @@ required packages. :: ) Another new enhancement to the Python package index at -https://pypi.python.org is storing source and binary archives for a +https://pypi.org is storing source and binary archives for a package. The new :command:`upload` Distutils command will upload a package to the repository. diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst index 53333eb..992658e 100644 --- a/Doc/whatsnew/2.7.rst +++ b/Doc/whatsnew/2.7.rst @@ -1218,11 +1218,6 @@ changes, or look through the Subversion logs for all the details. created some new files that should be included. (Fixed by Tarek Ziadé; :issue:`8688`.) - The ``upload`` command now longer tries to change CR end-of-line characters - to CRLF. This fixes a corruption issue with sdists that ended with a byte - equivalent to CR. - (Contributed by Bo Bayles in :issue:`32304`.) - * The :mod:`doctest` module's :const:`IGNORE_EXCEPTION_DETAIL` flag will now ignore the name of the module containing the exception being tested. (Patch by Lennart Regebro; :issue:`7490`.) @@ -1535,7 +1530,7 @@ changes, or look through the Subversion logs for all the details. *ciphers* argument that's a string listing the encryption algorithms to be allowed; the format of the string is described `in the OpenSSL documentation - <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__. + <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__. (Added by Antoine Pitrou; :issue:`8322`.) Another change makes the extension load all of OpenSSL's ciphers and @@ -1790,7 +1785,7 @@ wish to read the Tcl/Tk manual page describing the Ttk theme engine, available at https://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some screenshots of the Python/Ttk code in use are at -http://code.google.com/p/python-ttk/wiki/Screenshots. +https://code.google.com/archive/p/python-ttk/wikis/Screenshots.wiki. The :mod:`ttk` module was written by Guilherme Polo and added in :issue:`2983`. An alternate version called ``Tile.py``, written by @@ -1809,12 +1804,12 @@ new features were added. Most of these features were implemented by Michael Foord, unless otherwise noted. The enhanced version of the module is downloadable separately for use with Python versions 2.4 to 2.6, packaged as the :mod:`unittest2` package, from -https://pypi.python.org/pypi/unittest2. +https://pypi.org/project/unittest2. When used from the command line, the module can automatically discover tests. It's not as fancy as `py.test <http://pytest.org>`__ or -`nose <http://code.google.com/p/python-nose/>`__, but provides a simple way -to run tests kept within a set of package directories. For example, +`nose <https://nose.readthedocs.io/>`__, but provides a +simple way to run tests kept within a set of package directories. For example, the following command will search the :file:`test/` subdirectory for any importable test files named ``test*.py``:: @@ -2728,6 +2723,39 @@ For cases where the connection establishment code can't be modified, but the overall application can be, the new :func:`ssl._https_verify_certificates` function can be used to adjust the default behaviour at runtime. + +New ``make regen-all`` build target +----------------------------------- + +To simplify cross-compilation, and to ensure that CPython can reliably be +compiled without requiring an existing version of Python to already be +available, the autotools-based build system no longer attempts to implicitly +recompile generated files based on file modification times. + +Instead, a new ``make regen-all`` command has been added to force regeneration +of these files when desired (e.g. after an initial version of Python has +already been built based on the pregenerated versions). + +More selective regeneration targets are also defined - see +:source:`Makefile.pre.in` for details. + +(Contributed by Victor Stinner in :issue:`23404`.) + +.. versionadded:: 2.7.14 + + +Removal of ``make touch`` build target +-------------------------------------- + +The ``make touch`` build target previously used to request implicit regeneration +of generated files by updating their modification times has been removed. + +It has been replaced by the new ``make regen-all`` target. + +(Contributed by Victor Stinner in :issue:`23404`.) + +.. versionchanged:: 2.7.14 + .. ====================================================================== .. _acks27: diff --git a/Include/errcode.h b/Include/errcode.h index becec80..5c5a0f7 100644 --- a/Include/errcode.h +++ b/Include/errcode.h @@ -29,6 +29,7 @@ extern "C" { #define E_EOFS 23 /* EOF in triple-quoted string */ #define E_EOLS 24 /* EOL in single-quoted string */ #define E_LINECONT 25 /* Unexpected characters after a line continuation */ +#define E_IO 26 /* I/O error */ #ifdef __cplusplus } diff --git a/Include/frameobject.h b/Include/frameobject.h index 8439081..3460379 100644 --- a/Include/frameobject.h +++ b/Include/frameobject.h @@ -56,7 +56,7 @@ PyAPI_DATA(PyTypeObject) PyFrame_Type; #define PyFrame_Check(op) (Py_TYPE(op) == &PyFrame_Type) #define PyFrame_IsRestricted(f) \ - ((f)->f_builtins != (f)->f_tstate->interp->builtins) + ((f)->f_tstate && (f)->f_builtins != (f)->f_tstate->interp->builtins) PyAPI_FUNC(PyFrameObject *) PyFrame_New(PyThreadState *, PyCodeObject *, PyObject *, PyObject *); diff --git a/Include/grammar.h b/Include/grammar.h index 8426da3..decf2e6 100644 --- a/Include/grammar.h +++ b/Include/grammar.h @@ -69,6 +69,7 @@ typedef struct { /* FUNCTIONS */ grammar *newgrammar(int start); +void freegrammar(grammar *g); dfa *adddfa(grammar *g, int type, char *name); int addstate(dfa *d); void addarc(dfa *d, int from, int to, int lbl); diff --git a/Include/intobject.h b/Include/intobject.h index 59d0616..d198574 100644 --- a/Include/intobject.h +++ b/Include/intobject.h @@ -31,6 +31,9 @@ PyAPI_DATA(PyTypeObject) PyInt_Type; PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_INT_SUBCLASS) #define PyInt_CheckExact(op) (Py_TYPE(op) == &PyInt_Type) +#define _PyAnyInt_Check(op) (PyInt_Check(op) || PyLong_Check(op)) +#define _PyAnyInt_CheckExact(op) (PyInt_CheckExact(op) || PyLong_CheckExact(op)) + PyAPI_FUNC(PyObject *) PyInt_FromString(char*, char**, int); #ifdef Py_USING_UNICODE PyAPI_FUNC(PyObject *) PyInt_FromUnicode(Py_UNICODE*, Py_ssize_t, int); diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 99e8370..b12d901 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -22,12 +22,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 2 #define PY_MINOR_VERSION 7 -#define PY_MICRO_VERSION 15 +#define PY_MICRO_VERSION 16 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "2.7.15" +#define PY_VERSION "2.7.16" /*--end constants--*/ /* Subversion Revision number of this file (not of the repository). Empty diff --git a/Include/pgenheaders.h b/Include/pgenheaders.h index 2049ae3..4843de6 100644 --- a/Include/pgenheaders.h +++ b/Include/pgenheaders.h @@ -23,6 +23,7 @@ PyAPI_FUNC(void) PySys_WriteStderr(const char *format, ...) #define delbitset _Py_delbitset #define dumptree _Py_dumptree #define findlabel _Py_findlabel +#define freegrammar _Py_freegrammar #define mergebitset _Py_mergebitset #define meta_grammar _Py_meta_grammar #define newbitset _Py_newbitset diff --git a/Include/pyexpat.h b/Include/pyexpat.h index 5340ef5..3fc5fa5 100644 --- a/Include/pyexpat.h +++ b/Include/pyexpat.h @@ -3,7 +3,7 @@ /* note: you must import expat.h before importing this module! */ -#define PyExpat_CAPI_MAGIC "pyexpat.expat_CAPI 1.0" +#define PyExpat_CAPI_MAGIC "pyexpat.expat_CAPI 1.1" #define PyExpat_CAPSULE_NAME "pyexpat.expat_CAPI" struct PyExpat_CAPI @@ -43,6 +43,8 @@ struct PyExpat_CAPI XML_Parser parser, XML_UnknownEncodingHandler handler, void *encodingHandlerData); void (*SetUserData)(XML_Parser parser, void *userData); + /* might be none for expat < 2.1.0 */ + int (*SetHashSalt)(XML_Parser parser, unsigned long hash_salt); /* always add new stuff to the end! */ }; @@ -73,8 +73,8 @@ analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation; All -Rights Reserved" are retained in Python alone or in any derivative version +2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; +All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee. 3. In the event Licensee prepares a derivative work that is based on diff --git a/Lib/SocketServer.py b/Lib/SocketServer.py index 122430e..df56830 100644 --- a/Lib/SocketServer.py +++ b/Lib/SocketServer.py @@ -229,6 +229,9 @@ class BaseServer: # shutdown request and wastes cpu at all other times. r, w, e = _eintr_retry(select.select, [self], [], [], poll_interval) + # bpo-35017: shutdown() called during select(), exit immediately. + if self.__shutdown_request: + break if self in r: self._handle_request_noblock() finally: diff --git a/Lib/_pyio.py b/Lib/_pyio.py index f022a4e..98c2d58 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -1619,6 +1619,7 @@ class TextIOWrapper(TextIOBase): self.buffer.write(b) if self._line_buffering and (haslf or "\r" in s): self.flush() + self._set_decoded_chars('') self._snapshot = None if self._decoder: self._decoder.reset() diff --git a/Lib/argparse.py b/Lib/argparse.py index 1b233b8..d000486 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -324,7 +324,11 @@ class HelpFormatter(object): if len(prefix) + len(usage) > text_width: # break usage into wrappable parts - part_regexp = r'\(.*?\)+|\[.*?\]+|\S+' + part_regexp = ( + r'\(.*?\)+(?=\s|$)|' + r'\[.*?\]+(?=\s|$)|' + r'\S+' + ) opt_usage = format(optionals, groups) pos_usage = format(positionals, groups) opt_parts = _re.findall(part_regexp, opt_usage) diff --git a/Lib/bsddb/test/test_associate.py b/Lib/bsddb/test/test_associate.py index f0eadaa..4a8d2ae 100644 --- a/Lib/bsddb/test/test_associate.py +++ b/Lib/bsddb/test/test_associate.py @@ -114,6 +114,22 @@ class AssociateErrorTestCase(unittest.TestCase): dupDB.close() self.fail("DBError exception was expected") + @unittest.skipUnless(db.version() >= (4, 6), 'Needs 4.6+') + def test_associateListError(self): + db1 = db.DB(self.env) + db1.open('bad.db', "a.db", db.DB_BTREE, db.DB_CREATE) + db2 = db.DB(self.env) + db2.open('bad.db', "b.db", db.DB_BTREE, db.DB_CREATE) + + db1.associate(db2, lambda a, b: [0]) + + msg = "TypeError: The list returned by DB->associate callback" \ + " should be a list of strings." + with test_support.captured_output("stderr") as s: + db1.put("0", "1") + db1.close() + db2.close() + self.assertEquals(s.getvalue().strip(), msg) #---------------------------------------------------------------------- diff --git a/Lib/cProfile.py b/Lib/cProfile.py index b2efd04..cd53cc0 100755 --- a/Lib/cProfile.py +++ b/Lib/cProfile.py @@ -64,11 +64,11 @@ def help(): # ____________________________________________________________ class Profile(_lsprof.Profiler): - """Profile(custom_timer=None, time_unit=None, subcalls=True, builtins=True) + """Profile(timer=None, timeunit=None, subcalls=True, builtins=True) Builds a profiler object using the specified timer function. The default timer is a fast built-in one based on real time. - For custom timer functions returning integers, time_unit can + For custom timer functions returning integers, timeunit can be a float specifying a scale (i.e. how long each integer unit is, in seconds). """ @@ -161,7 +161,7 @@ def label(code): # ____________________________________________________________ def main(): - import os, sys + import os, sys, pstats from optparse import OptionParser usage = "cProfile.py [-o output_file_path] [-s sort] scriptfile [arg] ..." parser = OptionParser(usage=usage) @@ -170,7 +170,8 @@ def main(): help="Save stats to <outfile>", default=None) parser.add_option('-s', '--sort', dest="sort", help="Sort order when printing to stdout, based on pstats.Stats class", - default=-1) + default=-1, + choices=sorted(pstats.Stats.sort_arg_dict_default)) if not sys.argv[1:]: parser.print_usage() @@ -184,11 +184,12 @@ def parse_qs(qs, keep_blank_values=0, strict_parsing=0): return urlparse.parse_qs(qs, keep_blank_values, strict_parsing) -def parse_qsl(qs, keep_blank_values=0, strict_parsing=0): +def parse_qsl(qs, keep_blank_values=0, strict_parsing=0, max_num_fields=None): """Parse a query given as a string argument.""" warn("cgi.parse_qsl is deprecated, use urlparse.parse_qsl instead", PendingDeprecationWarning, 2) - return urlparse.parse_qsl(qs, keep_blank_values, strict_parsing) + return urlparse.parse_qsl(qs, keep_blank_values, strict_parsing, + max_num_fields) def parse_multipart(fp, pdict): """Parse multipart input. @@ -393,7 +394,8 @@ class FieldStorage: """ def __init__(self, fp=None, headers=None, outerboundary="", - environ=os.environ, keep_blank_values=0, strict_parsing=0): + environ=os.environ, keep_blank_values=0, strict_parsing=0, + max_num_fields=None): """Constructor. Read multipart/* until last part. Arguments, all optional: @@ -420,10 +422,14 @@ class FieldStorage: If false (the default), errors are silently ignored. If true, errors raise a ValueError exception. + max_num_fields: int. If set, then __init__ throws a ValueError + if there are more than n fields read by parse_qsl(). + """ method = 'GET' self.keep_blank_values = keep_blank_values self.strict_parsing = strict_parsing + self.max_num_fields = max_num_fields if 'REQUEST_METHOD' in environ: method = environ['REQUEST_METHOD'].upper() self.qs_on_post = None @@ -606,10 +612,9 @@ class FieldStorage: qs = self.fp.read(self.length) if self.qs_on_post: qs += '&' + self.qs_on_post - self.list = list = [] - for key, value in urlparse.parse_qsl(qs, self.keep_blank_values, - self.strict_parsing): - list.append(MiniFieldStorage(key, value)) + query = urlparse.parse_qsl(qs, self.keep_blank_values, + self.strict_parsing, self.max_num_fields) + self.list = [MiniFieldStorage(key, value) for key, value in query] self.skip_lines() FieldStorageClass = None @@ -621,19 +626,38 @@ class FieldStorage: raise ValueError, 'Invalid boundary in multipart form: %r' % (ib,) self.list = [] if self.qs_on_post: - for key, value in urlparse.parse_qsl(self.qs_on_post, - self.keep_blank_values, self.strict_parsing): - self.list.append(MiniFieldStorage(key, value)) + query = urlparse.parse_qsl(self.qs_on_post, + self.keep_blank_values, + self.strict_parsing, + self.max_num_fields) + self.list.extend(MiniFieldStorage(key, value) + for key, value in query) FieldStorageClass = None + # Propagate max_num_fields into the sub class appropriately + max_num_fields = self.max_num_fields + if max_num_fields is not None: + max_num_fields -= len(self.list) + klass = self.FieldStorageClass or self.__class__ part = klass(self.fp, {}, ib, - environ, keep_blank_values, strict_parsing) + environ, keep_blank_values, strict_parsing, + max_num_fields) + # Throw first part away while not part.done: headers = rfc822.Message(self.fp) part = klass(self.fp, headers, ib, - environ, keep_blank_values, strict_parsing) + environ, keep_blank_values, strict_parsing, + max_num_fields) + + if max_num_fields is not None: + max_num_fields -= 1 + if part.list: + max_num_fields -= len(part.list) + if max_num_fields < 0: + raise ValueError('Max number of fields exceeded') + self.list.append(part) self.skip_lines() diff --git a/Lib/cgitb.py b/Lib/cgitb.py index 8acc4b7..3689f8a 100644 --- a/Lib/cgitb.py +++ b/Lib/cgitb.py @@ -125,7 +125,7 @@ function calls leading up to the error, in the order they occurred.</p>''' args, varargs, varkw, locals = inspect.getargvalues(frame) call = '' if func != '?': - call = 'in ' + strong(func) + \ + call = 'in ' + strong(pydoc.html.escape(func)) + \ inspect.formatargvalues(args, varargs, varkw, locals, formatvalue=lambda value: '=' + pydoc.html.repr(value)) @@ -285,7 +285,7 @@ class Hook: if self.display: if plain: - doc = doc.replace('&', '&').replace('<', '<') + doc = pydoc.html.escape(doc) self.file.write('<pre>' + doc + '</pre>\n') else: self.file.write(doc + '\n') diff --git a/Lib/ctypes/test/test_arrays.py b/Lib/ctypes/test/test_arrays.py index 49aaab5..53859a3 100644 --- a/Lib/ctypes/test/test_arrays.py +++ b/Lib/ctypes/test/test_arrays.py @@ -1,4 +1,6 @@ import unittest +from test.support import precisionbigmemtest, _2G +import sys from ctypes import * from ctypes.test import need_symbol @@ -132,5 +134,10 @@ class ArrayTestCase(unittest.TestCase): t2 = my_int * 1 self.assertIs(t1, t2) + @unittest.skipUnless(sys.maxsize > 2**32, 'requires 64bit platform') + @precisionbigmemtest(size=_2G, memuse=1, dry_run=False) + def test_large_array(self, size): + a = c_char * size + if __name__ == '__main__': unittest.main() diff --git a/Lib/ctypes/test/test_as_parameter.py b/Lib/ctypes/test/test_as_parameter.py index f2fe10a..de730e9 100644 --- a/Lib/ctypes/test/test_as_parameter.py +++ b/Lib/ctypes/test/test_as_parameter.py @@ -24,7 +24,7 @@ class BasicWrapTestCase(unittest.TestCase): f.argtypes = [c_byte, c_wchar, c_int, c_long, c_float, c_double] result = f(self.wrap(1), self.wrap(u"x"), self.wrap(3), self.wrap(4), self.wrap(5.0), self.wrap(6.0)) self.assertEqual(result, 139) - self.assertTrue(type(result), int) + self.assertIs(type(result), int) def test_pointers(self): f = dll._testfunc_p_p diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py index bf894d4..db3d9e7 100644 --- a/Lib/ctypes/test/test_callbacks.py +++ b/Lib/ctypes/test/test_callbacks.py @@ -250,6 +250,7 @@ class SampleCallbacksTestCase(unittest.TestCase): def test_callback_large_struct(self): class Check: pass + # This should mirror the structure in Modules/_ctypes/_ctypes_test.c class X(Structure): _fields_ = [ ('first', c_ulong), @@ -261,6 +262,11 @@ class SampleCallbacksTestCase(unittest.TestCase): check.first = s.first check.second = s.second check.third = s.third + # See issue #29565. + # The structure should be passed by value, so + # any changes to it should not be reflected in + # the value passed + s.first = s.second = s.third = 0x0badf00d check = Check() s = X() @@ -281,6 +287,11 @@ class SampleCallbacksTestCase(unittest.TestCase): self.assertEqual(check.first, 0xdeadbeef) self.assertEqual(check.second, 0xcafebabe) self.assertEqual(check.third, 0x0bad1dea) + # See issue #29565. + # Ensure that the original struct is unchanged. + self.assertEqual(s.first, check.first) + self.assertEqual(s.second, check.second) + self.assertEqual(s.third, check.third) ################################################################ diff --git a/Lib/ctypes/test/test_strings.py b/Lib/ctypes/test/test_strings.py index 4b58e7c..879c58a 100644 --- a/Lib/ctypes/test/test_strings.py +++ b/Lib/ctypes/test/test_strings.py @@ -61,6 +61,13 @@ class StringArrayTestCase(unittest.TestCase): ## print BUF.from_param(c_char_p("python")) ## print BUF.from_param(BUF(*"pyth")) + def test_del_segfault(self): + BUF = c_char * 4 + buf = BUF() + with self.assertRaises(AttributeError): + del buf.raw + + @need_symbol('c_wchar') class WStringArrayTestCase(unittest.TestCase): def test(self): diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py index 5650189..9a863c9 100644 --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -3,6 +3,7 @@ from ctypes import * from ctypes.test import need_symbol from struct import calcsize import _testcapi +import _ctypes_test class SubclassesTest(unittest.TestCase): def test_subclass(self): @@ -401,6 +402,28 @@ class StructureTestCase(unittest.TestCase): (1, 0, 0, 0, 0, 0)) self.assertRaises(TypeError, lambda: Z(1, 2, 3, 4, 5, 6, 7)) + def test_pass_by_value(self): + # This should mirror the structure in Modules/_ctypes/_ctypes_test.c + class X(Structure): + _fields_ = [ + ('first', c_ulong), + ('second', c_ulong), + ('third', c_ulong), + ] + + s = X() + s.first = 0xdeadbeef + s.second = 0xcafebabe + s.third = 0x0bad1dea + dll = CDLL(_ctypes_test.__file__) + func = dll._testfunc_large_struct_update_value + func.argtypes = (X,) + func.restype = None + func(s) + self.assertEqual(s.first, 0xdeadbeef) + self.assertEqual(s.second, 0xcafebabe) + self.assertEqual(s.third, 0x0bad1dea) + class PointerMemberTestCase(unittest.TestCase): def test(self): diff --git a/Lib/ctypes/test/test_win32.py b/Lib/ctypes/test/test_win32.py index d22e139..13a9863 100644 --- a/Lib/ctypes/test/test_win32.py +++ b/Lib/ctypes/test/test_win32.py @@ -53,6 +53,24 @@ class FunctionCallTestCase(unittest.TestCase): windll.user32.GetDesktopWindow() @unittest.skipUnless(sys.platform == "win32", 'Windows-specific test') +class ReturnStructSizesTestCase(unittest.TestCase): + def test_sizes(self): + dll = CDLL(_ctypes_test.__file__) + for i in range(1, 11): + fields = [ ("f%d" % f, c_char) for f in range(1, i + 1)] + class S(Structure): + _fields_ = fields + f = getattr(dll, "TestSize%d" % i) + f.restype = S + res = f() + for i, f in enumerate(fields): + value = getattr(res, f[0]) + expected = chr(ord('a') + i) + self.assertEquals(value, expected) + + + +@unittest.skipUnless(sys.platform == "win32", 'Windows-specific test') class TestWintypes(unittest.TestCase): def test_HWND(self): from ctypes import wintypes diff --git a/Lib/curses/has_key.py b/Lib/curses/has_key.py index 1dd5a3b..60b7be9 100644 --- a/Lib/curses/has_key.py +++ b/Lib/curses/has_key.py @@ -182,7 +182,7 @@ if __name__ == '__main__': L = [] _curses.initscr() for key in _capability_names.keys(): - system = key in _curses + system = _curses.has_key(key) python = has_key(key) if system != python: L.append( 'Mismatch for key %s, system=%i, Python=%i' diff --git a/Lib/decimal.py b/Lib/decimal.py index e5329dd..220fa57 100644 --- a/Lib/decimal.py +++ b/Lib/decimal.py @@ -1909,7 +1909,7 @@ class Decimal(object): if not other and not self: return context._raise_error(InvalidOperation, 'at least one of pow() 1st argument ' - 'and 2nd argument must be nonzero ;' + 'and 2nd argument must be nonzero; ' '0**0 is not defined') # compute sign of result diff --git a/Lib/distutils/archive_util.py b/Lib/distutils/archive_util.py index 834b722..19a3bc4 100644 --- a/Lib/distutils/archive_util.py +++ b/Lib/distutils/archive_util.py @@ -162,7 +162,15 @@ def make_zipfile(base_name, base_dir, verbose=0, dry_run=0): zip = zipfile.ZipFile(zip_filename, "w", compression=zipfile.ZIP_DEFLATED) + if base_dir != os.curdir: + path = os.path.normpath(os.path.join(base_dir, '')) + zip.write(path, path) + log.info("adding '%s'", path) for dirpath, dirnames, filenames in os.walk(base_dir): + for name in dirnames: + path = os.path.normpath(os.path.join(dirpath, name, '')) + zip.write(path, path) + log.info("adding '%s'", path) for name in filenames: path = os.path.normpath(os.path.join(dirpath, name)) if os.path.isfile(path): diff --git a/Lib/distutils/command/bdist_dumb.py b/Lib/distutils/command/bdist_dumb.py index 2f3c668..d8e023d 100644 --- a/Lib/distutils/command/bdist_dumb.py +++ b/Lib/distutils/command/bdist_dumb.py @@ -35,7 +35,7 @@ class bdist_dumb (Command): ('skip-build', None, "skip rebuilding everything (for testing/debugging)"), ('relative', None, - "build the archive using relative paths" + "build the archive using relative paths " "(default: false)"), ('owner=', 'u', "Owner name used when creating a tar file" diff --git a/Lib/distutils/command/bdist_msi.py b/Lib/distutils/command/bdist_msi.py index 703f873..d2401bc 100644 --- a/Lib/distutils/command/bdist_msi.py +++ b/Lib/distutils/command/bdist_msi.py @@ -99,14 +99,14 @@ class bdist_msi (Command): ('no-target-compile', 'c', "do not compile .py to .pyc on the target system"), ('no-target-optimize', 'o', - "do not compile .py to .pyo (optimized)" + "do not compile .py to .pyo (optimized) " "on the target system"), ('dist-dir=', 'd', "directory to put final built distributions in"), ('skip-build', None, "skip rebuilding everything (for testing/debugging)"), ('install-script=', None, - "basename of installation script to be run after" + "basename of installation script to be run after " "installation or before deinstallation"), ('pre-install-script=', None, "Fully qualified filename of a script to be run before " diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py index 477e0ee..caadf48 100644 --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -63,7 +63,7 @@ class bdist_rpm (Command): "RPM \"vendor\" (eg. \"Joe Blow <joe@example.com>\") " "[default: maintainer or author from setup script]"), ('packager=', None, - "RPM packager (eg. \"Jane Doe <jane@example.net>\")" + "RPM packager (eg. \"Jane Doe <jane@example.net>\") " "[default: vendor]"), ('doc-files=', None, "list of documentation files (space or comma-separated)"), diff --git a/Lib/distutils/command/bdist_wininst.py b/Lib/distutils/command/bdist_wininst.py index aa9383a..a1de520 100644 --- a/Lib/distutils/command/bdist_wininst.py +++ b/Lib/distutils/command/bdist_wininst.py @@ -35,7 +35,7 @@ class bdist_wininst (Command): ('no-target-compile', 'c', "do not compile .py to .pyc on the target system"), ('no-target-optimize', 'o', - "do not compile .py to .pyo (optimized)" + "do not compile .py to .pyo (optimized) " "on the target system"), ('dist-dir=', 'd', "directory to put final built distributions in"), @@ -46,7 +46,7 @@ class bdist_wininst (Command): ('skip-build', None, "skip rebuilding everything (for testing/debugging)"), ('install-script=', None, - "basename of installation script to be run after" + "basename of installation script to be run after " "installation or before deinstallation"), ('pre-install-script=', None, "Fully qualified filename of a script to be run before " diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py index 2c68be3..86a85c1 100644 --- a/Lib/distutils/command/build_ext.py +++ b/Lib/distutils/command/build_ext.py @@ -366,7 +366,7 @@ class build_ext (Command): ext_name, build_info = ext log.warn(("old-style (ext_name, build_info) tuple found in " - "ext_modules for extension '%s'" + "ext_modules for extension '%s' " "-- please convert to Extension instance" % ext_name)) if not (isinstance(ext_name, str) and diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py index 321344a..737b293 100644 --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -208,7 +208,8 @@ def find_executable(executable, path=None): os.environ['PATH']. Returns the complete filename or None if not found. """ if path is None: - path = os.environ['PATH'] + path = os.environ.get('PATH', os.defpath) + paths = path.split(os.pathsep) base, ext = os.path.splitext(executable) diff --git a/Lib/distutils/tests/test_archive_util.py b/Lib/distutils/tests/test_archive_util.py index ed7c2ce..137100c 100644 --- a/Lib/distutils/tests/test_archive_util.py +++ b/Lib/distutils/tests/test_archive_util.py @@ -98,7 +98,7 @@ class ArchiveUtilTestCase(support.TempdirManager, try: names = tar.getnames() names.sort() - return tuple(names) + return names finally: tar.close() diff --git a/Lib/distutils/tests/test_bdist_dumb.py b/Lib/distutils/tests/test_bdist_dumb.py index 5db3a85..ef9e681 100644 --- a/Lib/distutils/tests/test_bdist_dumb.py +++ b/Lib/distutils/tests/test_bdist_dumb.py @@ -86,7 +86,7 @@ class BuildDumbTestCase(support.TempdirManager, finally: fp.close() - contents = sorted(os.path.basename(fn) for fn in contents) + contents = sorted(filter(None, map(os.path.basename, contents))) wanted = ['foo-0.1-py%s.%s.egg-info' % sys.version_info[:2], 'foo.py'] if not sys.dont_write_bytecode: wanted.append('foo.pyc') diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py index 81058b1..e94647f 100644 --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -8,6 +8,12 @@ from distutils.command.check import check, HAS_DOCUTILS from distutils.tests import support from distutils.errors import DistutilsSetupError +try: + import pygments +except ImportError: + pygments = None + + class CheckTestCase(support.LoggingSilencer, support.TempdirManager, unittest.TestCase): @@ -120,9 +126,15 @@ class CheckTestCase(support.LoggingSilencer, pkg_info, dist = self.create_dist(long_description=rest_with_code) cmd = check(dist) cmd.check_restructuredtext() - self.assertEqual(cmd._warnings, 0) msgs = cmd._check_rst_data(rest_with_code) - self.assertEqual(len(msgs), 0) + if pygments is not None: + self.assertEqual(len(msgs), 0) + else: + self.assertEqual(len(msgs), 1) + self.assertEqual( + str(msgs[0][1]), + 'Cannot analyze code. Pygments package not found.' + ) def test_check_all(self): diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py index 02c1d12..c503bd6 100644 --- a/Lib/distutils/tests/test_sdist.py +++ b/Lib/distutils/tests/test_sdist.py @@ -130,7 +130,9 @@ class SDistTestCase(PyPIRCCommandTestCase): zip_file.close() # making sure everything has been pruned correctly - self.assertEqual(len(content), 4) + expected = ['', 'PKG-INFO', 'README', 'setup.py', + 'somecode/', 'somecode/__init__.py'] + self.assertEqual(sorted(content), ['fake-1.0/' + x for x in expected]) @unittest.skipUnless(zlib, "requires zlib") def test_make_distribution(self): @@ -246,7 +248,13 @@ class SDistTestCase(PyPIRCCommandTestCase): zip_file.close() # making sure everything was added - self.assertEqual(len(content), 12) + expected = ['', 'PKG-INFO', 'README', 'buildout.cfg', + 'data/', 'data/data.dt', 'inroot.txt', + 'scripts/', 'scripts/script.py', 'setup.py', + 'some/', 'some/file.txt', 'some/other_file.txt', + 'somecode/', 'somecode/__init__.py', 'somecode/doc.dat', + 'somecode/doc.txt'] + self.assertEqual(sorted(content), ['fake-1.0/' + x for x in expected]) # checking the MANIFEST f = open(join(self.tmp_dir, 'MANIFEST')) diff --git a/Lib/distutils/tests/test_spawn.py b/Lib/distutils/tests/test_spawn.py index defa54d..061a72f 100644 --- a/Lib/distutils/tests/test_spawn.py +++ b/Lib/distutils/tests/test_spawn.py @@ -1,8 +1,11 @@ """Tests for distutils.spawn.""" -import unittest import os +import stat +import sys import time -from test.test_support import captured_stdout, run_unittest +import unittest +from test.support import captured_stdout, run_unittest +from test import support as test_support from distutils.spawn import _nt_quote_args from distutils.spawn import spawn, find_executable @@ -53,6 +56,48 @@ class SpawnTestCase(support.TempdirManager, os.chmod(exe, 0777) spawn([exe]) # should work without any error + def test_find_executable(self): + with test_support.temp_dir() as tmp_dir: + # use TESTFN to get a pseudo-unique filename + program_noeext = test_support.TESTFN + # Give the temporary program an ".exe" suffix for all. + # It's needed on Windows and not harmful on other platforms. + program = program_noeext + ".exe" + + filename = os.path.join(tmp_dir, program) + with open(filename, "wb"): + pass + os.chmod(filename, stat.S_IXUSR) + + # test path parameter + rv = find_executable(program, path=tmp_dir) + self.assertEqual(rv, filename) + + if sys.platform == 'win32': + # test without ".exe" extension + rv = find_executable(program_noeext, path=tmp_dir) + self.assertEqual(rv, filename) + + # test find in the current directory + with test_support.change_cwd(tmp_dir): + rv = find_executable(program) + self.assertEqual(rv, program) + + # test non-existent program + dont_exist_program = "dontexist_" + program + rv = find_executable(dont_exist_program , path=tmp_dir) + self.assertIsNone(rv) + + # test os.defpath: missing PATH environment variable + with test_support.EnvironmentVarGuard() as env: + from distutils import spawn + with test_support.swap_attr(spawn.os, 'defpath', tmp_dir): + env.pop('PATH') + + rv = find_executable(program) + self.assertEqual(rv, filename) + + def test_suite(): return unittest.makeSuite(SpawnTestCase) diff --git a/Lib/distutils/tests/test_util.py b/Lib/distutils/tests/test_util.py index 7898e07..e081709 100644 --- a/Lib/distutils/tests/test_util.py +++ b/Lib/distutils/tests/test_util.py @@ -1,11 +1,14 @@ """Tests for distutils.util.""" +import os import sys import unittest -from test.test_support import run_unittest +from test.test_support import run_unittest, swap_attr from distutils.errors import DistutilsByteCompileError from distutils.tests import support -from distutils.util import byte_compile, grok_environment_error +from distutils import util # used to patch _environ_checked +from distutils.util import (byte_compile, grok_environment_error, + check_environ, get_platform) class UtilTestCase(support.EnvironGuard, unittest.TestCase): @@ -26,6 +29,41 @@ class UtilTestCase(support.EnvironGuard, unittest.TestCase): msg = grok_environment_error(exc) self.assertEqual(msg, "error: Unable to find batch file") + def test_check_environ(self): + util._environ_checked = 0 + os.environ.pop('HOME', None) + + check_environ() + + self.assertEqual(os.environ['PLAT'], get_platform()) + self.assertEqual(util._environ_checked, 1) + + @unittest.skipUnless(os.name == 'posix', 'specific to posix') + def test_check_environ_getpwuid(self): + util._environ_checked = 0 + os.environ.pop('HOME', None) + + import pwd + + # only set pw_dir field, other fields are not used + def mock_getpwuid(uid): + return pwd.struct_passwd((None, None, None, None, None, + '/home/distutils', None)) + + with swap_attr(pwd, 'getpwuid', mock_getpwuid): + check_environ() + self.assertEqual(os.environ['HOME'], '/home/distutils') + + util._environ_checked = 0 + os.environ.pop('HOME', None) + + # bpo-10496: Catch pwd.getpwuid() error + def getpwuid_err(uid): + raise KeyError + with swap_attr(pwd, 'getpwuid', getpwuid_err): + check_environ() + self.assertNotIn('HOME', os.environ) + def test_suite(): return unittest.makeSuite(UtilTestCase) diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py index 2b4d784..5a06b85 100644 --- a/Lib/distutils/util.py +++ b/Lib/distutils/util.py @@ -178,8 +178,13 @@ def check_environ (): return if os.name == 'posix' and 'HOME' not in os.environ: - import pwd - os.environ['HOME'] = pwd.getpwuid(os.getuid())[5] + try: + import pwd + os.environ['HOME'] = pwd.getpwuid(os.getuid())[5] + except (ImportError, KeyError): + # bpo-10496: if the current user identifier doesn't exist in the + # password database, do nothing + pass if 'PLAT' not in os.environ: os.environ['PLAT'] = get_platform() diff --git a/Lib/doctest.py b/Lib/doctest.py index fedf670..1d822b5 100644 --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -1651,8 +1651,6 @@ class OutputChecker: kind = 'ndiff with -expected +actual' else: assert 0, 'Bad diff option' - # Remove trailing whitespace on diff output. - diff = [line.rstrip() + '\n' for line in diff] return 'Differences (%s):\n' % kind + _indent(''.join(diff)) # If we're not using diff, then simply list the expected diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index 89ed1ef..5021ebf 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,9 +12,9 @@ import tempfile __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "39.0.1" +_SETUPTOOLS_VERSION = "40.6.2" -_PIP_VERSION = "9.0.3" +_PIP_VERSION = "18.1" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION), @@ -28,8 +28,8 @@ def _run_pip(args, additional_paths=None): sys.path = additional_paths + sys.path # Install the bundled software - import pip - return pip.main(args) + import pip._internal + return pip._internal.main(args) def version(): @@ -190,8 +190,8 @@ def _main(argv=None): "--altinstall", action="store_true", default=False, - help=("Make an alternate install, installing only the X.Y versioned" - "scripts (Default: pipX, pipX.Y, easy_install-X.Y)"), + help=("Make an alternate install, installing only the X.Y versioned " + "scripts (Default: pipX, pipX.Y, easy_install-X.Y)."), ) parser.add_argument( "--default-pip", diff --git a/Lib/ensurepip/_bundled/pip-18.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-18.1-py2.py3-none-any.whl Binary files differnew file mode 100644 index 0000000..c3c146f --- /dev/null +++ b/Lib/ensurepip/_bundled/pip-18.1-py2.py3-none-any.whl diff --git a/Lib/ensurepip/_bundled/pip-9.0.3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-9.0.3-py2.py3-none-any.whl Binary files differdeleted file mode 100644 index 5d33bba..0000000 --- a/Lib/ensurepip/_bundled/pip-9.0.3-py2.py3-none-any.whl +++ /dev/null diff --git a/Lib/ensurepip/_bundled/setuptools-39.0.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-40.6.2-py2.py3-none-any.whl Binary files differindex edc3ca2..4c8a619 100644 --- a/Lib/ensurepip/_bundled/setuptools-39.0.1-py2.py3-none-any.whl +++ b/Lib/ensurepip/_bundled/setuptools-40.6.2-py2.py3-none-any.whl diff --git a/Lib/gzip.py b/Lib/gzip.py index 07c6db4..76ace39 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -95,9 +95,8 @@ class GzipFile(io.BufferedIOBase): if filename is None: # Issue #13781: os.fdopen() creates a fileobj with a bogus name # attribute. Avoid saving this in the gzip header's filename field. - if hasattr(fileobj, 'name') and fileobj.name != '<fdopen>': - filename = fileobj.name - else: + filename = getattr(fileobj, 'name', '') + if not isinstance(filename, basestring) or filename == '<fdopen>': filename = '' if mode is None: if hasattr(fileobj, 'mode'): mode = fileobj.mode diff --git a/Lib/httplib.py b/Lib/httplib.py index f3bb22c..60a8fb4 100644 --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -399,7 +399,7 @@ class HTTPResponse: if not line: # Presumably, the server closed the connection before # sending a valid response. - raise BadStatusLine(line) + raise BadStatusLine("No status line received - the server has closed the connection") try: [version, status, reason] = line.split(None, 2) except ValueError: diff --git a/Lib/idlelib/CallTipWindow.py b/Lib/idlelib/CallTipWindow.py index e63164b..2a453d0 100644 --- a/Lib/idlelib/CallTipWindow.py +++ b/Lib/idlelib/CallTipWindow.py @@ -72,6 +72,7 @@ class CallTip: background="#ffffe0", relief=SOLID, borderwidth=1, font = self.widget['font']) self.label.pack() + tw.update_idletasks() tw.lift() # work around bug in Tk 8.5.18+ (issue #24570) self.checkhideid = self.widget.bind(CHECKHIDE_VIRTUAL_EVENT_NAME, diff --git a/Lib/idlelib/FileList.py b/Lib/idlelib/FileList.py index 8318ff1..46979e3 100644 --- a/Lib/idlelib/FileList.py +++ b/Lib/idlelib/FileList.py @@ -107,8 +107,10 @@ class FileList: def _test(): from idlelib.EditorWindow import fixwordbreaks + from idlelib.run import fix_scaling import sys root = Tk() + fix_scaling(root) fixwordbreaks(root) root.withdraw() flist = FileList(root) diff --git a/Lib/idlelib/HyperParser.py b/Lib/idlelib/HyperParser.py index 5816d00..6e45b16 100644 --- a/Lib/idlelib/HyperParser.py +++ b/Lib/idlelib/HyperParser.py @@ -167,7 +167,7 @@ class HyperParser: given index, which is empty if there is no real one. """ if not self.is_in_code(): - raise ValueError("get_expression should only be called" + raise ValueError("get_expression should only be called " "if index is inside a code.") rawtext = self.rawtext diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 35e2a7e..7eb30a1 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,21 @@ +Since 2.7.13, only severe bugs are fixed on the 2.7 branch. + +What's New in IDLE 2.7.16? +========================== +*Release date: 2019-01-01?* + +bpo-31500: Default fonts now are scaled on HiDPI displays. + +bpo-34275: Make calltips always visible on Mac. +Patch by Kevin Walzer. + +bpo-34120: Fix freezing after closing some dialogs on Mac. +This is one of multiple regressions from using newer tcl/tk. + + What's New in IDLE 2.7.13? ========================== -*Release date: 2017-01-01?* +*Release date: 2016-12-17* - Issue #27854: Make Help => IDLE Help work again on Windows. Include idlelib/help.html in 2.7 Windows installer. @@ -35,7 +50,7 @@ What's New in IDLE 2.7.13? What's New in IDLE 2.7.12? ========================== -*Release date: 2015-06-25* +*Release date: 2016-06-25* - Issue #5124: Paste with text selected now replaces the selection on X11. This matches how paste works on Windows, Mac, most modern Linux apps, diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py index fdd7cb1..2ea7e6b 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1040,7 +1040,7 @@ class PyShell(OutputWindow): return self.shell_title COPYRIGHT = \ - 'Type "copyright", "credits" or "license()" for more information.' + 'Type "help", "copyright", "credits" or "license()" for more information.' def begin(self): self.resetoutput() @@ -1552,13 +1552,15 @@ def main(): # start editor and/or shell windows: root = Tk(className="Idle") root.withdraw() + from idlelib.run import fix_scaling + fix_scaling(root) # set application icon icondir = os.path.join(os.path.dirname(__file__), 'Icons') if system() == 'Windows': iconfile = os.path.join(icondir, 'idle.ico') root.wm_iconbitmap(default=iconfile) - elif TkVersion >= 8.5: + elif TkVersion >= 8.5 and sys.platform != 'darwin': ext = '.png' if TkVersion >= 8.6 else '.gif' iconfiles = [os.path.join(icondir, 'idle_%d%s' % (size, ext)) for size in (16, 32, 48)] diff --git a/Lib/idlelib/aboutDialog.py b/Lib/idlelib/aboutDialog.py index c9adc08..87d6c3c 100644 --- a/Lib/idlelib/aboutDialog.py +++ b/Lib/idlelib/aboutDialog.py @@ -141,6 +141,7 @@ class AboutDialog(Toplevel): textView.view_file(self, title, fn, encoding) def Ok(self, event=None): + self.grab_release() self.destroy() if __name__ == '__main__': diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py index d53f5ff..3c29af1 100644 --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -1197,10 +1197,12 @@ class ConfigDialog(Toplevel): instance.reset_help_menu_entries() def Cancel(self): + self.grab_release() self.destroy() def Ok(self): self.Apply() + self.grab_release() self.destroy() def Apply(self): diff --git a/Lib/idlelib/configHelpSourceEdit.py b/Lib/idlelib/configHelpSourceEdit.py index 5816449..62b010a 100644 --- a/Lib/idlelib/configHelpSourceEdit.py +++ b/Lib/idlelib/configHelpSourceEdit.py @@ -155,10 +155,12 @@ class GetHelpSourceDialog(Toplevel): # Mac Safari insists on using the URI form for local files self.result = list(self.result) self.result[1] = "file://" + path + self.grab_release() self.destroy() def Cancel(self, event=None): self.result = None + self.grab_release() self.destroy() if __name__ == '__main__': diff --git a/Lib/idlelib/configSectionNameDialog.py b/Lib/idlelib/configSectionNameDialog.py index c09dca8..f28dc1a 100644 --- a/Lib/idlelib/configSectionNameDialog.py +++ b/Lib/idlelib/configSectionNameDialog.py @@ -80,10 +80,13 @@ class GetCfgSectionNameDialog(Toplevel): name = self.name_ok() if name: self.result = name + self.grab_release() self.destroy() def Cancel(self, event=None): self.result = '' + self.grab_release() self.destroy() + if __name__ == '__main__': import unittest unittest.main('idlelib.idle_test.test_config_name', verbosity=2, exit=False) diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py index f341409..9e2ddd2 100644 --- a/Lib/idlelib/idle_test/htest.py +++ b/Lib/idlelib/idle_test/htest.py @@ -112,7 +112,7 @@ ConfigDialog_spec = { "font face of the text in the area below it.\nIn the " "'Highlighting' tab, try different color schemes. Clicking " "items in the sample program should update the choices above it." - "\nIn the 'Keys', 'General' and 'Extensions' tabs, test settings" + "\nIn the 'Keys', 'General' and 'Extensions' tabs, test settings " "of interest." "\n[Ok] to close the dialog.[Apply] to apply the settings and " "and [Cancel] to revert all changes.\nRe-run the test to ensure " @@ -171,7 +171,7 @@ GetKeysDialog_spec = { 'msg': "Test for different key modifier sequences.\n" "<nothing> is invalid.\n" "No modifier key is invalid.\n" - "Shift key with [a-z],[0-9], function key, move key, tab, space" + "Shift key with [a-z],[0-9], function key, move key, tab, space " "is invalid.\nNo validitity checking if advanced key binding " "entry is used." } @@ -237,7 +237,7 @@ _percolator_spec = { 'file': 'Percolator', 'kwds': {}, 'msg': "There are two tracers which can be toggled using a checkbox.\n" - "Toggling a tracer 'on' by checking it should print tracer" + "Toggling a tracer 'on' by checking it should print tracer " "output to the console or to the IDLE shell.\n" "If both the tracers are 'on', the output from the tracer which " "was switched 'on' later, should be printed first\n" @@ -329,7 +329,7 @@ _undo_delegator_spec = { _widget_redirector_spec = { 'file': 'WidgetRedirector', 'kwds': {}, - 'msg': "Every text insert should be printed to the console." + 'msg': "Every text insert should be printed to the console " "or the IDLE shell." } diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py index f42a039..56ca876 100644 --- a/Lib/idlelib/idle_test/mock_tk.py +++ b/Lib/idlelib/idle_test/mock_tk.py @@ -260,7 +260,7 @@ class Text(object): elif op == '!=': return line1 != line2 or char1 != char2 else: - raise TclError('''bad comparison operator "%s":''' + raise TclError('''bad comparison operator "%s": ''' '''must be <, <=, ==, >=, >, or !=''' % op) # The following Text methods normally do something and return None. diff --git a/Lib/idlelib/idle_test/test_config_name.py b/Lib/idlelib/idle_test/test_config_name.py index 4403f87..2a4df6a 100644 --- a/Lib/idlelib/idle_test/test_config_name.py +++ b/Lib/idlelib/idle_test/test_config_name.py @@ -15,6 +15,8 @@ class Dummy_name_dialog(object): name = Var() result = None destroyed = False + def grab_release(self): + pass def destroy(self): self.destroyed = True diff --git a/Lib/idlelib/keybindingDialog.py b/Lib/idlelib/keybindingDialog.py index 4d32ca9..755f1af 100644 --- a/Lib/idlelib/keybindingDialog.py +++ b/Lib/idlelib/keybindingDialog.py @@ -217,10 +217,12 @@ class GetKeysDialog(Toplevel): def OK(self, event=None): if self.advanced or self.KeysOK(): # doesn't check advanced string yet self.result=self.keyString.get() + self.grab_release() self.destroy() def Cancel(self, event=None): self.result='' + self.grab_release() self.destroy() def KeysOK(self): diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py index 466c61e..518afab 100644 --- a/Lib/idlelib/run.py +++ b/Lib/idlelib/run.py @@ -155,6 +155,7 @@ def show_socket_error(err, address): import Tkinter import tkMessageBox root = Tkinter.Tk() + fix_scaling(root) root.withdraw() if err.args[0] == 61: # connection refused msg = "IDLE's subprocess can't connect to %s:%d. This may be due "\ @@ -240,6 +241,19 @@ def exit(): capture_warnings(False) sys.exit(0) + +def fix_scaling(root): + """Scale fonts on HiDPI displays.""" + import tkFont + scaling = float(root.tk.call('tk', 'scaling')) + if scaling > 1.4: + for name in tkFont.names(root): + font = tkFont.Font(root=root, name=name, exists=True) + size = int(font['size']) + if size < 0: + font['size'] = int(round(-0.75*size)) + + class MyRPCServer(rpc.RPCServer): def handle_error(self, request, client_address): diff --git a/Lib/idlelib/textView.py b/Lib/idlelib/textView.py index b8c4ac1..ec837f8 100644 --- a/Lib/idlelib/textView.py +++ b/Lib/idlelib/textView.py @@ -39,7 +39,8 @@ class TextViewer(Toplevel): self.textView.insert(0.0, text) self.textView.config(state=DISABLED) - if modal: + self.is_modal = modal + if self.is_modal: self.transient(parent) self.grab_set() self.wait_window() @@ -62,6 +63,8 @@ class TextViewer(Toplevel): frameText.pack(side=TOP,expand=TRUE,fill=BOTH) def Ok(self, event=None): + if self.is_modal: + self.grab_release() self.destroy() diff --git a/Lib/imaplib.py b/Lib/imaplib.py index 220d6e1..679c468 100644 --- a/Lib/imaplib.py +++ b/Lib/imaplib.py @@ -70,6 +70,7 @@ Commands = { 'LOGIN': ('NONAUTH',), 'LOGOUT': ('NONAUTH', 'AUTH', 'SELECTED', 'LOGOUT'), 'LSUB': ('AUTH', 'SELECTED'), + 'MOVE': ('SELECTED',), 'NAMESPACE': ('AUTH', 'SELECTED'), 'NOOP': ('NONAUTH', 'AUTH', 'SELECTED', 'LOGOUT'), 'PARTIAL': ('SELECTED',), # NB: obsolete @@ -1181,16 +1182,6 @@ else: self.file = self.sslobj.makefile('rb') - def read(self, size): - """Read 'size' bytes from remote.""" - return self.file.read(size) - - - def readline(self): - """Read line from remote.""" - return self.file.readline() - - def send(self, data): """Send data to remote.""" bytes = len(data) diff --git a/Lib/inspect.py b/Lib/inspect.py index 0a6cfd7..cbced17 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -688,8 +688,15 @@ def getsourcelines(object): raised if the source code cannot be retrieved.""" lines, lnum = findsource(object) - if ismodule(object): return lines, 0 - else: return getblock(lines[lnum:]), lnum + 1 + if istraceback(object): + object = object.tb_frame + + # for module or frame that corresponds to module, return all source lines + if (ismodule(object) or + (isframe(object) and object.f_code.co_name == "<module>")): + return lines, 0 + else: + return getblock(lines[lnum:]), lnum + 1 def getsource(object): """Return the text of the source code for an object. diff --git a/Lib/lib-tk/Tkinter.py b/Lib/lib-tk/Tkinter.py index 6198c4c..6c02955 100644 --- a/Lib/lib-tk/Tkinter.py +++ b/Lib/lib-tk/Tkinter.py @@ -71,7 +71,7 @@ def _stringify(value): if isinstance(value, (list, tuple)): if len(value) == 1: value = _stringify(value[0]) - if value[0] == '{': + if _magic_re.search(value): value = '{%s}' % value else: value = '{%s}' % _join(value) @@ -85,7 +85,10 @@ def _stringify(value): elif _magic_re.search(value): # add '\' before special characters and spaces value = _magic_re.sub(r'\\\1', value) + value = value.replace('\n', r'\n') value = _space_re.sub(r'\\\1', value) + if value[0] == '"': + value = '\\' + value elif value[0] == '"' or _space_re.search(value): value = '{%s}' % value return value @@ -586,6 +589,7 @@ class Misc: if not func: # I'd rather use time.sleep(ms*0.001) self.tk.call('after', ms) + return None else: def callit(): try: @@ -609,11 +613,13 @@ class Misc: """Cancel scheduling of function identified with ID. Identifier returned by after or after_idle must be - given as first parameter.""" + given as first parameter. + """ + if not id: + raise ValueError('id must be a valid identifier returned from ' + 'after or after_idle') try: data = self.tk.call('after', 'info', id) - # In Tk 8.3, splitlist returns: (script, type) - # In Tk 8.4, splitlist may return (script, type) or (script,) script = self.tk.splitlist(data)[0] self.deletecommand(script) except TclError: @@ -3362,7 +3368,7 @@ class Image: self.tk.call('image', 'width', self.name)) class PhotoImage(Image): - """Widget which can display colored images in GIF, PPM/PGM format.""" + """Widget which can display images in PGM, PPM, GIF, PNG format.""" def __init__(self, name=None, cnf={}, master=None, **kw): """Create an image with NAME. @@ -3426,7 +3432,7 @@ class PhotoImage(Image): self.tk.call(args) class BitmapImage(Image): - """Widget which can display a bitmap.""" + """Widget which can display images in XBM format.""" def __init__(self, name=None, cnf={}, master=None, **kw): """Create a bitmap with NAME. @@ -3576,7 +3582,7 @@ class Spinbox(Widget, XView): select to commands. If the selection isn't currently in the spinbox, then a new selection is created to include the characters between index and the most recent selection - anchor point, inclusive. Returns an empty string. + anchor point, inclusive. """ return self.selection("adjust", index) @@ -3584,7 +3590,7 @@ class Spinbox(Widget, XView): """Clear the selection If the selection isn't in this widget then the - command has no effect. Returns an empty string. + command has no effect. """ return self.selection("clear") @@ -3592,9 +3598,9 @@ class Spinbox(Widget, XView): """Sets or gets the currently selected element. If a spinbutton element is specified, it will be - displayed depressed + displayed depressed. """ - return self.selection("element", element) + return self.tk.call(self._w, 'selection', 'element', element) ########################################################################### diff --git a/Lib/lib-tk/test/test_tkinter/test_misc.py b/Lib/lib-tk/test/test_tkinter/test_misc.py new file mode 100644 index 0000000..796269e --- /dev/null +++ b/Lib/lib-tk/test/test_tkinter/test_misc.py @@ -0,0 +1,122 @@ +import unittest +import Tkinter as tkinter +from test.test_support import requires, run_unittest +from test_ttk.support import AbstractTkTest + +requires('gui') + +class MiscTest(AbstractTkTest, unittest.TestCase): + + def test_after(self): + root = self.root + cbcount = {'count': 0} + + def callback(start=0, step=1): + cbcount['count'] = start + step + + # Without function, sleeps for ms. + self.assertIsNone(root.after(1)) + + # Set up with callback with no args. + cbcount['count'] = 0 + timer1 = root.after(0, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.update() # Process all pending events. + self.assertEqual(cbcount['count'], 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + cbcount['count'] = 0 + timer1 = root.after(0, callback, 42, 11) + root.update() # Process all pending events. + self.assertEqual(cbcount['count'], 53) + + # Cancel before called. + timer1 = root.after(1000, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.after_cancel(timer1) # Cancel this event. + self.assertEqual(cbcount['count'], 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_idle(self): + root = self.root + cbcount = {'count': 0} + + def callback(start=0, step=1): + cbcount['count'] = start + step + + # Set up with callback with no args. + cbcount['count'] = 0 + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.update_idletasks() # Process all pending events. + self.assertEqual(cbcount['count'], 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + cbcount['count'] = 0 + idle1 = root.after_idle(callback, 42, 11) + root.update_idletasks() # Process all pending events. + self.assertEqual(cbcount['count'], 53) + + # Cancel before called. + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.after_cancel(idle1) # Cancel this event. + self.assertEqual(cbcount['count'], 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_cancel(self): + root = self.root + cbcount = {'count': 0} + + def callback(): + cbcount['count'] += 1 + + timer1 = root.after(5000, callback) + idle1 = root.after_idle(callback) + + # No value for id raises a ValueError. + with self.assertRaises(ValueError): + root.after_cancel(None) + + # Cancel timer event. + cbcount['count'] = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.tk.call(script) + self.assertEqual(cbcount['count'], 1) + root.after_cancel(timer1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(cbcount['count'], 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', timer1) + + # Cancel same event - nothing happens. + root.after_cancel(timer1) + + # Cancel idle event. + cbcount['count'] = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.tk.call(script) + self.assertEqual(cbcount['count'], 1) + root.after_cancel(idle1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(cbcount['count'], 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', idle1) + + +tests_gui = (MiscTest, ) + +if __name__ == "__main__": + run_unittest(*tests_gui) diff --git a/Lib/lib-tk/test/test_tkinter/test_widgets.py b/Lib/lib-tk/test/test_tkinter/test_widgets.py index 1f60204..4b196ac 100644 --- a/Lib/lib-tk/test/test_tkinter/test_widgets.py +++ b/Lib/lib-tk/test/test_tkinter/test_widgets.py @@ -471,6 +471,14 @@ class SpinboxTest(EntryTest, unittest.TestCase): self.assertRaises(TypeError, widget.bbox) self.assertRaises(TypeError, widget.bbox, 0, 1) + def test_selection_element(self): + widget = self.create() + self.assertEqual(widget.selection_element(), "none") + widget.selection_element("buttonup") + self.assertEqual(widget.selection_element(), "buttonup") + widget.selection_element("buttondown") + self.assertEqual(widget.selection_element(), "buttondown") + @add_standard_options(StandardOptionsTests) class TextTest(AbstractWidgetTest, unittest.TestCase): @@ -700,7 +708,7 @@ class ListboxTest(AbstractWidgetTest, unittest.TestCase): 'disabledforeground', 'exportselection', 'font', 'foreground', 'height', 'highlightbackground', 'highlightcolor', 'highlightthickness', - 'listvariable', 'relief', + 'justify', 'listvariable', 'relief', 'selectbackground', 'selectborderwidth', 'selectforeground', 'selectmode', 'setgrid', 'state', 'takefocus', 'width', 'xscrollcommand', 'yscrollcommand', @@ -714,6 +722,8 @@ class ListboxTest(AbstractWidgetTest, unittest.TestCase): self.checkEnumParam(widget, 'activestyle', 'dotbox', 'none', 'underline') + test_justify = requires_tcl(8, 6, 5)(StandardOptionsTests.test_justify.im_func) + def test_listvariable(self): widget = self.create() var = tkinter.DoubleVar(self.root) @@ -947,7 +957,9 @@ class PanedWindowTest(AbstractWidgetTest, unittest.TestCase): OPTIONS = ( 'background', 'borderwidth', 'cursor', 'handlepad', 'handlesize', 'height', - 'opaqueresize', 'orient', 'relief', + 'opaqueresize', 'orient', + 'proxybackground', 'proxyborderwidth', 'proxyrelief', + 'relief', 'sashcursor', 'sashpad', 'sashrelief', 'sashwidth', 'showhandle', 'width', ) @@ -974,6 +986,23 @@ class PanedWindowTest(AbstractWidgetTest, unittest.TestCase): widget = self.create() self.checkBooleanParam(widget, 'opaqueresize') + @requires_tcl(8, 6, 5) + def test_proxybackground(self): + widget = self.create() + self.checkColorParam(widget, 'proxybackground') + + @requires_tcl(8, 6, 5) + def test_proxyborderwidth(self): + widget = self.create() + self.checkPixelsParam(widget, 'proxyborderwidth', + 0, 1.3, 2.9, 6, -2, '10p', + conv=noconv) + + @requires_tcl(8, 6, 5) + def test_proxyrelief(self): + widget = self.create() + self.checkReliefParam(widget, 'proxyrelief') + def test_sashcursor(self): widget = self.create() self.checkCursorParam(widget, 'sashcursor') diff --git a/Lib/lib-tk/test/test_ttk/support.py b/Lib/lib-tk/test/test_ttk/support.py index c4d842a..a86e0ea 100644 --- a/Lib/lib-tk/test/test_ttk/support.py +++ b/Lib/lib-tk/test/test_ttk/support.py @@ -1,3 +1,4 @@ +import functools import re import unittest import Tkinter as tkinter @@ -54,9 +55,20 @@ import _tkinter tcl_version = tuple(map(int, _tkinter.TCL_VERSION.split('.'))) def requires_tcl(*version): - return unittest.skipUnless(tcl_version >= version, + if len(version) <= 2: + return unittest.skipUnless(tcl_version >= version, 'requires Tcl version >= ' + '.'.join(map(str, version))) + def deco(test): + @functools.wraps(test) + def newtest(self): + if get_tk_patchlevel() < version: + self.skipTest('requires Tcl version >= ' + + '.'.join(map(str, version))) + test(self) + return newtest + return deco + _tk_patchlevel = None def get_tk_patchlevel(): global _tk_patchlevel diff --git a/Lib/lib-tk/turtle.py b/Lib/lib-tk/turtle.py index 52e669b..ae921ce 100644 --- a/Lib/lib-tk/turtle.py +++ b/Lib/lib-tk/turtle.py @@ -1300,7 +1300,7 @@ class TurtleScreen(TurtleScreenBase): Arguments: fun -- a function with two arguments, the coordinates of the clicked point on the canvas. - num -- the number of the mouse-button, defaults to 1 + btn -- the number of the mouse-button, defaults to 1 Example (for a TurtleScreen instance named screen and a Turtle instance named turtle): @@ -3418,7 +3418,7 @@ class RawTurtle(TPen, TNavigator): Arguments: fun -- a function with two arguments, to which will be assigned the coordinates of the clicked point on the canvas. - num -- number of the mouse-button defaults to 1 (left mouse button). + btn -- number of the mouse-button defaults to 1 (left mouse button). add -- True or False. If True, new binding will be added, otherwise it will replace a former binding. @@ -3439,7 +3439,7 @@ class RawTurtle(TPen, TNavigator): Arguments: fun -- a function with two arguments, to which will be assigned the coordinates of the clicked point on the canvas. - num -- number of the mouse-button defaults to 1 (left mouse button). + btn -- number of the mouse-button defaults to 1 (left mouse button). Example (for a MyTurtle instance named joe): >>> class MyTurtle(Turtle): @@ -3464,7 +3464,7 @@ class RawTurtle(TPen, TNavigator): Arguments: fun -- a function with two arguments, to which will be assigned the coordinates of the clicked point on the canvas. - num -- number of the mouse-button defaults to 1 (left mouse button). + btn -- number of the mouse-button defaults to 1 (left mouse button). Every sequence of mouse-move-events on a turtle is preceded by a mouse-click event on that turtle. diff --git a/Lib/lib2to3/fixes/fix_execfile.py b/Lib/lib2to3/fixes/fix_execfile.py index 2f29d3b..786268b 100644 --- a/Lib/lib2to3/fixes/fix_execfile.py +++ b/Lib/lib2to3/fixes/fix_execfile.py @@ -31,7 +31,8 @@ class FixExecfile(fixer_base.BaseFix): # call. execfile_paren = node.children[-1].children[-1].clone() # Construct open().read(). - open_args = ArgList([filename.clone()], rparen=execfile_paren) + open_args = ArgList([filename.clone(), Comma(), String('"rb"', ' ')], + rparen=execfile_paren) open_call = Node(syms.power, [Name(u"open"), open_args]) read = [Node(syms.trailer, [Dot(), Name(u'read')]), Node(syms.trailer, [LParen(), RParen()])] diff --git a/Lib/lib2to3/tests/test_fixers.py b/Lib/lib2to3/tests/test_fixers.py index b0e60fe..c7d5ff9 100644 --- a/Lib/lib2to3/tests/test_fixers.py +++ b/Lib/lib2to3/tests/test_fixers.py @@ -1143,36 +1143,36 @@ class Test_execfile(FixerTestCase): def test_conversion(self): b = """execfile("fn")""" - a = """exec(compile(open("fn").read(), "fn", 'exec'))""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'))""" self.check(b, a) b = """execfile("fn", glob)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), glob)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), glob)""" self.check(b, a) b = """execfile("fn", glob, loc)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), glob, loc)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), glob, loc)""" self.check(b, a) b = """execfile("fn", globals=glob)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), globals=glob)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), globals=glob)""" self.check(b, a) b = """execfile("fn", locals=loc)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), locals=loc)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), locals=loc)""" self.check(b, a) b = """execfile("fn", globals=glob, locals=loc)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), globals=glob, locals=loc)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), globals=glob, locals=loc)""" self.check(b, a) def test_spacing(self): b = """execfile( "fn" )""" - a = """exec(compile(open( "fn" ).read(), "fn", 'exec'))""" + a = """exec(compile(open( "fn", "rb" ).read(), "fn", 'exec'))""" self.check(b, a) b = """execfile("fn", globals = glob)""" - a = """exec(compile(open("fn").read(), "fn", 'exec'), globals = glob)""" + a = """exec(compile(open("fn", "rb").read(), "fn", 'exec'), globals = glob)""" self.check(b, a) diff --git a/Lib/locale.py b/Lib/locale.py index 62aad5a..51909f8 100644 --- a/Lib/locale.py +++ b/Lib/locale.py @@ -798,6 +798,17 @@ locale_encoding_alias = { # # SS 2014-10-01: # Updated alias mapping with glibc 2.19 supported locales. +# +# SS 2018-05-05: +# Updated alias mapping with glibc 2.27 supported locales. +# +# These are the differences compared to the old mapping (Python 2.7.15 +# and older): +# +# updated 'ca_es@valencia' -> 'ca_ES.ISO8859-15@valencia' to 'ca_ES.UTF-8@valencia' +# updated 'english.iso88591' -> 'en_EN.ISO8859-1' to 'en_US.ISO8859-1' +# updated 'kk_kz' -> 'kk_KZ.RK1048' to 'kk_KZ.ptcp154' +# updated 'russian' -> 'ru_RU.ISO8859-5' to 'ru_RU.KOI8-R' locale_alias = { 'a3': 'az_AZ.KOI8-C', @@ -810,11 +821,14 @@ locale_alias = { 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', 'af_za.iso88591': 'af_ZA.ISO8859-1', + 'agr_pe': 'agr_PE.UTF-8', + 'ak_gh': 'ak_GH.UTF-8', 'am': 'am_ET.UTF-8', 'am_et': 'am_ET.UTF-8', 'american': 'en_US.ISO8859-1', 'american.iso88591': 'en_US.ISO8859-1', 'an_es': 'an_ES.ISO8859-15', + 'anp_in': 'anp_IN.UTF-8', 'ar': 'ar_AA.ISO8859-6', 'ar_aa': 'ar_AA.ISO8859-6', 'ar_aa.iso88596': 'ar_AA.ISO8859-6', @@ -847,6 +861,7 @@ locale_alias = { 'ar_sa.iso88596': 'ar_SA.ISO8859-6', 'ar_sd': 'ar_SD.ISO8859-6', 'ar_sd.iso88596': 'ar_SD.ISO8859-6', + 'ar_ss': 'ar_SS.UTF-8', 'ar_sy': 'ar_SY.ISO8859-6', 'ar_sy.iso88596': 'ar_SY.ISO8859-6', 'ar_tn': 'ar_TN.ISO8859-6', @@ -862,6 +877,7 @@ locale_alias = { 'az': 'az_AZ.ISO8859-9E', 'az_az': 'az_AZ.ISO8859-9E', 'az_az.iso88599e': 'az_AZ.ISO8859-9E', + 'az_ir': 'az_IR.UTF-8', 'be': 'be_BY.CP1251', 'be@latin': 'be_BY.UTF-8@latin', 'be_bg.utf8': 'bg_BG.UTF-8', @@ -879,7 +895,10 @@ locale_alias = { 'bg_bg.iso88595': 'bg_BG.ISO8859-5', 'bg_bg.koi8r': 'bg_BG.KOI8-R', 'bg_bg.microsoftcp1251': 'bg_BG.CP1251', + 'bhb_in.utf8': 'bhb_IN.UTF-8', 'bho_in': 'bho_IN.UTF-8', + 'bho_np': 'bho_NP.UTF-8', + 'bi_vu': 'bi_VU.UTF-8', 'bn_bd': 'bn_BD.UTF-8', 'bn_in': 'bn_IN.UTF-8', 'bo_cn': 'bo_CN.UTF-8', @@ -921,8 +940,8 @@ locale_alias = { 'ca_es.iso885915': 'ca_ES.ISO8859-15', 'ca_es.iso885915@euro': 'ca_ES.ISO8859-15', 'ca_es.utf8@euro': 'ca_ES.UTF-8', - 'ca_es@valencia': 'ca_ES.ISO8859-15@valencia', 'ca_es@euro': 'ca_ES.ISO8859-15', + 'ca_es@valencia': 'ca_ES.UTF-8@valencia', 'ca_fr': 'ca_FR.ISO8859-1', 'ca_fr.iso88591': 'ca_FR.ISO8859-1', 'ca_fr.iso885915': 'ca_FR.ISO8859-15', @@ -936,10 +955,14 @@ locale_alias = { 'ca_it.utf8@euro': 'ca_IT.UTF-8', 'ca_it@euro': 'ca_IT.ISO8859-15', 'catalan': 'ca_ES.ISO8859-1', + 'ce_ru': 'ce_RU.UTF-8', 'cextend': 'en_US.ISO8859-1', 'cextend.en': 'en_US.ISO8859-1', 'chinese-s': 'zh_CN.eucCN', 'chinese-t': 'zh_TW.eucTW', + 'chr_us': 'chr_US.UTF-8', + 'ckb_iq': 'ckb_IQ.UTF-8', + 'cmn_tw': 'cmn_TW.UTF-8', 'crh_ua': 'crh_UA.UTF-8', 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', @@ -996,6 +1019,7 @@ locale_alias = { 'de_de.iso885915@euro': 'de_DE.ISO8859-15', 'de_de.utf8@euro': 'de_DE.UTF-8', 'de_de@euro': 'de_DE.ISO8859-15', + 'de_it': 'de_IT.ISO8859-1', 'de_li.utf8': 'de_LI.UTF-8', 'de_lu': 'de_LU.ISO8859-1', 'de_lu.iso88591': 'de_LU.ISO8859-1', @@ -1030,6 +1054,8 @@ locale_alias = { 'en_ca': 'en_CA.ISO8859-1', 'en_ca.iso88591': 'en_CA.ISO8859-1', 'en_dk': 'en_DK.ISO8859-1', + 'en_dk.iso88591': 'en_DK.ISO8859-1', + 'en_dk.iso885915': 'en_DK.ISO8859-15', 'en_dl.utf8': 'en_DL.UTF-8', 'en_gb': 'en_GB.ISO8859-1', 'en_gb.88591': 'en_GB.ISO8859-1', @@ -1044,12 +1070,14 @@ locale_alias = { 'en_ie.iso885915@euro': 'en_IE.ISO8859-15', 'en_ie.utf8@euro': 'en_IE.UTF-8', 'en_ie@euro': 'en_IE.ISO8859-15', + 'en_il': 'en_IL.UTF-8', 'en_in': 'en_IN.ISO8859-1', 'en_ng': 'en_NG.UTF-8', 'en_nz': 'en_NZ.ISO8859-1', 'en_nz.iso88591': 'en_NZ.ISO8859-1', 'en_ph': 'en_PH.ISO8859-1', 'en_ph.iso88591': 'en_PH.ISO8859-1', + 'en_sc.utf8': 'en_SC.UTF-8', 'en_sg': 'en_SG.ISO8859-1', 'en_sg.iso88591': 'en_SG.ISO8859-1', 'en_uk': 'en_GB.ISO8859-1', @@ -1073,7 +1101,7 @@ locale_alias = { 'eng_gb': 'en_GB.ISO8859-1', 'eng_gb.8859': 'en_GB.ISO8859-1', 'english': 'en_EN.ISO8859-1', - 'english.iso88591': 'en_EN.ISO8859-1', + 'english.iso88591': 'en_US.ISO8859-1', 'english_uk': 'en_GB.ISO8859-1', 'english_uk.8859': 'en_GB.ISO8859-1', 'english_united-states': 'en_US.ISO8859-1', @@ -1268,6 +1296,7 @@ locale_alias = { 'gv_gb.iso885915': 'gv_GB.ISO8859-15', 'gv_gb@euro': 'gv_GB.ISO8859-15', 'ha_ng': 'ha_NG.UTF-8', + 'hak_tw': 'hak_TW.UTF-8', 'he': 'he_IL.ISO8859-8', 'he_il': 'he_IL.ISO8859-8', 'he_il.cp1255': 'he_IL.CP1255', @@ -1278,6 +1307,7 @@ locale_alias = { 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', + 'hif_fj': 'hif_FJ.UTF-8', 'hne': 'hne_IN.UTF-8', 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', @@ -1361,7 +1391,8 @@ locale_alias = { 'ka_ge.georgianacademy': 'ka_GE.GEORGIAN-ACADEMY', 'ka_ge.georgianps': 'ka_GE.GEORGIAN-PS', 'ka_ge.georgianrs': 'ka_GE.GEORGIAN-ACADEMY', - 'kk_kz': 'kk_KZ.RK1048', + 'kab_dz': 'kab_DZ.UTF-8', + 'kk_kz': 'kk_KZ.ptcp154', 'kl': 'kl_GL.ISO8859-1', 'kl_gl': 'kl_GL.ISO8859-1', 'kl_gl.iso88591': 'kl_GL.ISO8859-1', @@ -1379,6 +1410,7 @@ locale_alias = { 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', 'ks_in': 'ks_IN.UTF-8', + 'ks_in.utf8@devanagari': 'ks_IN.UTF-8@devanagari', 'ks_in@devanagari': 'ks_IN.UTF-8@devanagari', 'ks_in@devanagari.utf8': 'ks_IN.UTF-8@devanagari', 'ku_tr': 'ku_TR.ISO8859-9', @@ -1396,6 +1428,7 @@ locale_alias = { 'li_nl': 'li_NL.UTF-8', 'lij_it': 'lij_IT.UTF-8', 'lithuanian': 'lt_LT.ISO8859-13', + 'ln_cd': 'ln_CD.UTF-8', 'lo': 'lo_LA.MULELAO-1', 'lo_la': 'lo_LA.MULELAO-1', 'lo_la.cp1133': 'lo_LA.IBM-CP1133', @@ -1409,14 +1442,19 @@ locale_alias = { 'lv_lv': 'lv_LV.ISO8859-13', 'lv_lv.iso885913': 'lv_LV.ISO8859-13', 'lv_lv.iso88594': 'lv_LV.ISO8859-4', + 'lzh_tw': 'lzh_TW.UTF-8', 'mag_in': 'mag_IN.UTF-8', 'mai': 'mai_IN.UTF-8', 'mai_in': 'mai_IN.UTF-8', + 'mai_np': 'mai_NP.UTF-8', + 'mfe_mu': 'mfe_MU.UTF-8', 'mg_mg': 'mg_MG.ISO8859-15', 'mhr_ru': 'mhr_RU.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', 'mi_nz.iso88591': 'mi_NZ.ISO8859-1', + 'miq_ni': 'miq_NI.UTF-8', + 'mjw_in': 'mjw_IN.UTF-8', 'mk': 'mk_MK.ISO8859-5', 'mk_mk': 'mk_MK.ISO8859-5', 'mk_mk.cp1251': 'mk_MK.CP1251', @@ -1435,7 +1473,7 @@ locale_alias = { 'mt_mt': 'mt_MT.ISO8859-3', 'mt_mt.iso88593': 'mt_MT.ISO8859-3', 'my_mm': 'my_MM.UTF-8', - 'nan_tw@latin': 'nan_TW.UTF-8@latin', + 'nan_tw': 'nan_TW.UTF-8', 'nb': 'nb_NO.ISO8859-1', 'nb_no': 'nb_NO.ISO8859-1', 'nb_no.88591': 'nb_NO.ISO8859-1', @@ -1509,6 +1547,8 @@ locale_alias = { 'pa_in': 'pa_IN.UTF-8', 'pa_pk': 'pa_PK.UTF-8', 'pap_an': 'pap_AN.UTF-8', + 'pap_aw': 'pap_AW.UTF-8', + 'pap_cw': 'pap_CW.UTF-8', 'pd': 'pd_US.ISO8859-1', 'pd_de': 'pd_DE.ISO8859-1', 'pd_de.iso88591': 'pd_DE.ISO8859-1', @@ -1549,6 +1589,8 @@ locale_alias = { 'pt_pt.iso885915@euro': 'pt_PT.ISO8859-15', 'pt_pt.utf8@euro': 'pt_PT.UTF-8', 'pt_pt@euro': 'pt_PT.ISO8859-15', + 'quz_pe': 'quz_PE.UTF-8', + 'raj_in': 'raj_IN.UTF-8', 'ro': 'ro_RO.ISO8859-2', 'ro_ro': 'ro_RO.ISO8859-2', 'ro_ro.iso88592': 'ro_RO.ISO8859-2', @@ -1565,7 +1607,7 @@ locale_alias = { 'ru_ua.koi8u': 'ru_UA.KOI8-U', 'ru_ua.microsoftcp1251': 'ru_UA.CP1251', 'rumanian': 'ro_RO.ISO8859-2', - 'russian': 'ru_RU.ISO8859-5', + 'russian': 'ru_RU.KOI8-R', 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', @@ -1575,17 +1617,20 @@ locale_alias = { 'sd': 'sd_IN.UTF-8', 'sd@devanagari': 'sd_IN.UTF-8@devanagari', 'sd_in': 'sd_IN.UTF-8', + 'sd_in.utf8@devanagari': 'sd_IN.UTF-8@devanagari', 'sd_in@devanagari': 'sd_IN.UTF-8@devanagari', 'sd_in@devanagari.utf8': 'sd_IN.UTF-8@devanagari', 'sd_pk': 'sd_PK.UTF-8', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8@latin', + 'sgs_lt': 'sgs_LT.UTF-8', 'sh': 'sr_RS.UTF-8@latin', 'sh_ba.iso88592@bosnia': 'sr_CS.ISO8859-2', 'sh_hr': 'sh_HR.ISO8859-2', 'sh_hr.iso88592': 'hr_HR.ISO8859-2', 'sh_sp': 'sr_CS.ISO8859-2', 'sh_yu': 'sr_RS.UTF-8@latin', + 'shn_mm': 'shn_MM.UTF-8', 'shs_ca': 'shs_CA.UTF-8', 'si': 'si_LK.UTF-8', 'si_lk': 'si_LK.UTF-8', @@ -1601,6 +1646,7 @@ locale_alias = { 'slovak': 'sk_SK.ISO8859-2', 'slovene': 'sl_SI.ISO8859-2', 'slovenian': 'sl_SI.ISO8859-2', + 'sm_ws': 'sm_WS.UTF-8', 'so_dj': 'so_DJ.ISO8859-1', 'so_et': 'so_ET.UTF-8', 'so_ke': 'so_KE.ISO8859-1', @@ -1627,6 +1673,7 @@ locale_alias = { 'sr_cs@latn': 'sr_CS.UTF-8@latin', 'sr_me': 'sr_ME.UTF-8', 'sr_rs': 'sr_RS.UTF-8', + 'sr_rs.utf8@latn': 'sr_RS.UTF-8@latin', 'sr_rs@latin': 'sr_RS.UTF-8@latin', 'sr_rs@latn': 'sr_RS.UTF-8@latin', 'sr_sp': 'sr_CS.ISO8859-2', @@ -1668,6 +1715,7 @@ locale_alias = { 'ta_in.tscii': 'ta_IN.TSCII-0', 'ta_in.tscii0': 'ta_IN.TSCII-0', 'ta_lk': 'ta_LK.UTF-8', + 'tcy_in.utf8': 'tcy_IN.UTF-8', 'te': 'te_IN.UTF-8', 'te_in': 'te_IN.UTF-8', 'tg': 'tg_TJ.KOI8-C', @@ -1679,6 +1727,7 @@ locale_alias = { 'th_th.tactis': 'th_TH.TIS620', 'th_th.tis620': 'th_TH.TIS620', 'thai': 'th_TH.ISO8859-11', + 'the_np': 'the_NP.UTF-8', 'ti_er': 'ti_ER.UTF-8', 'ti_et': 'ti_ET.UTF-8', 'tig_er': 'tig_ER.UTF-8', @@ -1689,6 +1738,8 @@ locale_alias = { 'tn': 'tn_ZA.ISO8859-15', 'tn_za': 'tn_ZA.ISO8859-15', 'tn_za.iso885915': 'tn_ZA.ISO8859-15', + 'to_to': 'to_TO.UTF-8', + 'tpi_pg': 'tpi_PG.UTF-8', 'tr': 'tr_TR.ISO8859-9', 'tr_cy': 'tr_CY.ISO8859-9', 'tr_tr': 'tr_TR.ISO8859-9', @@ -1750,6 +1801,7 @@ locale_alias = { 'yi_us.microsoftcp1255': 'yi_US.CP1255', 'yo_ng': 'yo_NG.UTF-8', 'yue_hk': 'yue_HK.UTF-8', + 'yuw_pg': 'yuw_PG.UTF-8', 'zh': 'zh_CN.eucCN', 'zh_cn': 'zh_CN.gb2312', 'zh_cn.big5': 'zh_TW.big5', diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py index caecc11..afc053f 100644 --- a/Lib/mimetypes.py +++ b/Lib/mimetypes.py @@ -451,6 +451,7 @@ def _default_mime_types(): '.mht' : 'message/rfc822', '.mhtml' : 'message/rfc822', '.mif' : 'application/x-mif', + '.mjs' : 'application/javascript', '.mov' : 'video/quicktime', '.movie' : 'video/x-sgi-movie', '.mp2' : 'audio/mpeg', diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py index f2cee0c..118812c 100644 --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -1059,10 +1059,13 @@ class ListProxy(BaseListProxy): DictProxy = MakeProxyType('DictProxy', ( - '__contains__', '__delitem__', '__getitem__', '__len__', + '__contains__', '__delitem__', '__getitem__', '__iter__', '__len__', '__setitem__', 'clear', 'copy', 'get', 'has_key', 'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values' )) +DictProxy._method_to_typeid_ = { + '__iter__': 'Iterator', + } ArrayProxy = MakeProxyType('ArrayProxy', ( diff --git a/Lib/pickletools.py b/Lib/pickletools.py index c9366c8..cc15540 100644 --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -1048,9 +1048,7 @@ opcodes = [ stack_before=[], stack_after=[pybool], proto=2, - doc="""True. - - Push True onto the stack."""), + doc="Push True onto the stack."), I(name='NEWFALSE', code='\x89', @@ -1058,9 +1056,7 @@ opcodes = [ stack_before=[], stack_after=[pybool], proto=2, - doc="""True. - - Push False onto the stack."""), + doc="Push False onto the stack."), # Ways to spell Unicode strings. diff --git a/Lib/platform.py b/Lib/platform.py index 55f2fa8..e04d87f 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -132,6 +132,35 @@ except AttributeError: # Standard Unix uses /dev/null DEV_NULL = '/dev/null' +# Helper for comparing two version number strings. +# Based on the description of the PHP's version_compare(): +# http://php.net/manual/en/function.version-compare.php + +_ver_stages = { + # any string not found in this dict, will get 0 assigned + 'dev': 10, + 'alpha': 20, 'a': 20, + 'beta': 30, 'b': 30, + 'c': 40, + 'RC': 50, 'rc': 50, + # number, will get 100 assigned + 'pl': 200, 'p': 200, +} + +_component_re = re.compile(r'([0-9]+|[._+-])') + +def _comparable_version(version): + result = [] + for v in _component_re.split(version): + if v not in '._+-': + try: + v = int(v, 10) + t = 100 + except ValueError: + t = _ver_stages.get(v, 0) + result.extend((t, v)) + return result + ### Platform specific APIs _libc_search = re.compile(r'(__libc_init)' @@ -140,9 +169,7 @@ _libc_search = re.compile(r'(__libc_init)' '|' '(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)') -def libc_ver(executable=sys.executable,lib='',version='', - - chunksize=2048): +def libc_ver(executable=sys.executable,lib='',version='', chunksize=2048): """ Tries to determine the libc version that the file executable (which defaults to the Python interpreter) is linked against. @@ -157,40 +184,45 @@ def libc_ver(executable=sys.executable,lib='',version='', The file is read and scanned in chunks of chunksize bytes. """ + V = _comparable_version if hasattr(os.path, 'realpath'): # Python 2.2 introduced os.path.realpath(); it is used # here to work around problems with Cygwin not being # able to open symlinks for reading executable = os.path.realpath(executable) - f = open(executable,'rb') - binary = f.read(chunksize) - pos = 0 - while 1: - m = _libc_search.search(binary,pos) - if not m: - binary = f.read(chunksize) - if not binary: - break - pos = 0 - continue - libcinit,glibc,glibcversion,so,threads,soversion = m.groups() - if libcinit and not lib: - lib = 'libc' - elif glibc: - if lib != 'glibc': - lib = 'glibc' - version = glibcversion - elif glibcversion > version: - version = glibcversion - elif so: - if lib != 'glibc': + with open(executable, 'rb') as f: + binary = f.read(chunksize) + pos = 0 + while pos < len(binary): + if 'libc' in binary or 'GLIBC' in binary: + m = _libc_search.search(binary, pos) + else: + m = None + if not m or m.end() == len(binary): + chunk = f.read(chunksize) + if chunk: + binary = binary[max(pos, len(binary) - 1000):] + chunk + pos = 0 + continue + if not m: + break + libcinit,glibc,glibcversion,so,threads,soversion = m.groups() + if libcinit and not lib: lib = 'libc' - if soversion and soversion > version: - version = soversion - if threads and version[-len(threads):] != threads: - version = version + threads - pos = m.end() - f.close() + elif glibc: + if lib != 'glibc': + lib = 'glibc' + version = glibcversion + elif V(glibcversion) > V(version): + version = glibcversion + elif so: + if lib != 'glibc': + lib = 'libc' + if soversion and (not version or V(soversion) > V(version)): + version = soversion + if threads and version[-len(threads):] != threads: + version = version + threads + pos = m.end() return lib,version def _dist_try_harder(distname,version,id): @@ -451,6 +483,7 @@ def popen(cmd, mode='r', bufsize=None): else: return popen(cmd,mode,bufsize) + def _norm_version(version, build=''): """ Normalize the version and build strings and return a single diff --git a/Lib/posixpath.py b/Lib/posixpath.py index f5c2260..bbc2369 100644 --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -259,7 +259,12 @@ def expanduser(path): if i == 1: if 'HOME' not in os.environ: import pwd - userhome = pwd.getpwuid(os.getuid()).pw_dir + try: + userhome = pwd.getpwuid(os.getuid()).pw_dir + except KeyError: + # bpo-10496: if the current user identifier doesn't exist in the + # password database, return the path unchanged + return path else: userhome = os.environ['HOME'] else: @@ -267,6 +272,8 @@ def expanduser(path): try: pwent = pwd.getpwnam(path[1:i]) except KeyError: + # bpo-10496: if the user name from the path doesn't exist in the + # password database, return the path unchanged return path userhome = pwent.pw_dir userhome = userhome.rstrip('/') diff --git a/Lib/pydoc.py b/Lib/pydoc.py index b4b190f..62cc262 100755 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -1647,8 +1647,9 @@ class Helper: } # Either add symbols to this dictionary or to the symbols dictionary # directly: Whichever is easier. They are merged later. + _strprefixes = tuple(p + q for p in ('b', 'r', 'u') for q in ("'", '"')) _symbols_inverse = { - 'STRINGS' : ("'", "'''", "r'", "u'", '"""', '"', 'r"', 'u"'), + 'STRINGS' : ("'", "'''", '"""', '"') + _strprefixes, 'OPERATORS' : ('+', '-', '*', '**', '/', '//', '%', '<<', '>>', '&', '|', '^', '~', '<', '>', '<=', '>=', '==', '!=', '<>'), 'COMPARISON' : ('<', '>', '<=', '>=', '==', '!=', '<>'), @@ -1811,7 +1812,12 @@ has the same effect as typing a particular string at the help> prompt. if not request: break except (KeyboardInterrupt, EOFError): break - request = strip(replace(request, '"', '', "'", '')) + request = strip(request) + # Make sure significant trailing quotation marks of literals don't + # get deleted while cleaning input + if (len(request) > 2 and request[0] == request[-1] in ("'", '"') + and request[0] not in request[1:-1]): + request = request[1:-1] if lower(request) in ('q', 'quit'): break self.help(request) diff --git a/Lib/robotparser.py b/Lib/robotparser.py index a7137a3..4e13f7f 100644 --- a/Lib/robotparser.py +++ b/Lib/robotparser.py @@ -160,7 +160,10 @@ class RobotFileParser: def __str__(self): - return ''.join([str(entry) + "\n" for entry in self.entries]) + entries = self.entries + if self.default_entry is not None: + entries = entries + [self.default_entry] + return '\n'.join(map(str, entries)) + '\n' class RuleLine: diff --git a/Lib/shutil.py b/Lib/shutil.py index 3462f7c..a45436c 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -105,7 +105,13 @@ def copymode(src, dst): os.chmod(dst, mode) def copystat(src, dst): - """Copy all stat info (mode bits, atime, mtime, flags) from src to dst""" + """Copy file metadata + + Copy the permission bits, last access time, last modification time, and + flags from `src` to `dst`. On Linux, copystat() also copies the "extended + attributes" where possible. The file contents, owner, and group are + unaffected. `src` and `dst` are path names given as strings. + """ st = os.stat(src) mode = stat.S_IMODE(st.st_mode) if hasattr(os, 'utime'): @@ -134,7 +140,10 @@ def copy(src, dst): copymode(src, dst) def copy2(src, dst): - """Copy data and all stat info ("cp -p src dst"). + """Copy data and metadata. Return the file's destination. + + Metadata is copied with copystat(). Please see the copystat function + for more information. The destination may be a directory. @@ -413,17 +422,21 @@ def _make_tarball(base_name, base_dir, compress="gzip", verbose=0, dry_run=0, return archive_name -def _call_external_zip(base_dir, zip_filename, verbose=False, dry_run=False): +def _call_external_zip(base_dir, zip_filename, verbose, dry_run, logger): # XXX see if we want to keep an external call here if verbose: zipoptions = "-r" else: zipoptions = "-rq" - from distutils.errors import DistutilsExecError - from distutils.spawn import spawn + cmd = ["zip", zipoptions, zip_filename, base_dir] + if logger is not None: + logger.info(' '.join(cmd)) + if dry_run: + return + import subprocess try: - spawn(["zip", zipoptions, zip_filename, base_dir], dry_run=dry_run) - except DistutilsExecError: + subprocess.check_call(cmd) + except subprocess.CalledProcessError: # XXX really should distinguish between "couldn't find # external 'zip' command" and "zip failed". raise ExecError, \ @@ -458,7 +471,7 @@ def _make_zipfile(base_name, base_dir, verbose=0, dry_run=0, logger=None): zipfile = None if zipfile is None: - _call_external_zip(base_dir, zip_filename, verbose, dry_run) + _call_external_zip(base_dir, zip_filename, verbose, dry_run, logger) else: if logger is not None: logger.info("creating '%s' and adding '%s' to it", diff --git a/Lib/sqlite3/test/regression.py b/Lib/sqlite3/test/regression.py index 7eeac32..42fc7c2 100644 --- a/Lib/sqlite3/test/regression.py +++ b/Lib/sqlite3/test/regression.py @@ -246,24 +246,6 @@ class RegressionTests(unittest.TestCase): cur.execute("pragma page_size") row = cur.fetchone() - def CheckSetDict(self): - """ - See http://bugs.python.org/issue7478 - - It was possible to successfully register callbacks that could not be - hashed. Return codes of PyDict_SetItem were not checked properly. - """ - class NotHashable: - def __call__(self, *args, **kw): - pass - def __hash__(self): - raise TypeError() - var = NotHashable() - self.assertRaises(TypeError, self.con.create_function, var) - self.assertRaises(TypeError, self.con.create_aggregate, var) - self.assertRaises(TypeError, self.con.set_authorizer, var) - self.assertRaises(TypeError, self.con.set_progress_handler, var) - def CheckConnectionCall(self): """ Call a connection with a non-string SQL request: check error handling @@ -379,10 +361,78 @@ class RegressionTests(unittest.TestCase): del ref support.gc_collect() + def CheckDelIsolation_levelSegfault(self): + with self.assertRaises(AttributeError): + del self.con.isolation_level + + +class UnhashableFunc: + def __hash__(self): + raise TypeError('unhashable type') + + def __init__(self, return_value=None): + self.calls = 0 + self.return_value = return_value + + def __call__(self, *args, **kwargs): + self.calls += 1 + return self.return_value + + +class UnhashableCallbacksTestCase(unittest.TestCase): + """ + https://bugs.python.org/issue34052 + + Registering unhashable callbacks raises TypeError, callbacks are not + registered in SQLite after such registration attempt. + """ + def setUp(self): + self.con = sqlite.connect(':memory:') + + def tearDown(self): + self.con.close() + + def test_progress_handler(self): + f = UnhashableFunc(return_value=0) + with self.assertRaisesRegexp(TypeError, 'unhashable type'): + self.con.set_progress_handler(f, 1) + self.con.execute('SELECT 1') + self.assertFalse(f.calls) + + def test_func(self): + func_name = 'func_name' + f = UnhashableFunc() + with self.assertRaisesRegexp(TypeError, 'unhashable type'): + self.con.create_function(func_name, 0, f) + msg = 'no such function: %s' % func_name + with self.assertRaisesRegexp(sqlite.OperationalError, msg): + self.con.execute('SELECT %s()' % func_name) + self.assertFalse(f.calls) + + def test_authorizer(self): + f = UnhashableFunc(return_value=sqlite.SQLITE_DENY) + with self.assertRaisesRegexp(TypeError, 'unhashable type'): + self.con.set_authorizer(f) + self.con.execute('SELECT 1') + self.assertFalse(f.calls) + + def test_aggr(self): + class UnhashableType(type): + __hash__ = None + aggr_name = 'aggr_name' + with self.assertRaisesRegexp(TypeError, 'unhashable type'): + self.con.create_aggregate(aggr_name, 0, UnhashableType('Aggr', (), {})) + msg = 'no such function: %s' % aggr_name + with self.assertRaisesRegexp(sqlite.OperationalError, msg): + self.con.execute('SELECT %s()' % aggr_name) + def suite(): regression_suite = unittest.makeSuite(RegressionTests, "Check") - return unittest.TestSuite((regression_suite,)) + return unittest.TestSuite(( + regression_suite, + unittest.makeSuite(UnhashableCallbacksTestCase), + )) def test(): runner = unittest.TextTestRunner() @@ -424,32 +424,16 @@ def create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None, if not isinstance(purpose, _ASN1Object): raise TypeError(purpose) + # SSLContext sets OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION, + # OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE and OP_SINGLE_ECDH_USE + # by default. context = SSLContext(PROTOCOL_TLS) - # SSLv2 considered harmful. - context.options |= OP_NO_SSLv2 - - # SSLv3 has problematic security and is only required for really old - # clients such as IE6 on Windows XP - context.options |= OP_NO_SSLv3 - - # disable compression to prevent CRIME attacks (OpenSSL 1.0+) - context.options |= getattr(_ssl, "OP_NO_COMPRESSION", 0) - if purpose == Purpose.SERVER_AUTH: # verify certs and host name in client mode context.verify_mode = CERT_REQUIRED context.check_hostname = True elif purpose == Purpose.CLIENT_AUTH: - # Prefer the server's ciphers by default so that we get stronger - # encryption - context.options |= getattr(_ssl, "OP_CIPHER_SERVER_PREFERENCE", 0) - - # Use single use keys in order to improve forward secrecy - context.options |= getattr(_ssl, "OP_SINGLE_DH_USE", 0) - context.options |= getattr(_ssl, "OP_SINGLE_ECDH_USE", 0) - - # disallow ciphers with known vulnerabilities context.set_ciphers(_RESTRICTED_SERVER_CIPHERS) if cafile or capath or cadata: @@ -475,12 +459,10 @@ def _create_unverified_context(protocol=PROTOCOL_TLS, cert_reqs=None, if not isinstance(purpose, _ASN1Object): raise TypeError(purpose) + # SSLContext sets OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION, + # OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE and OP_SINGLE_ECDH_USE + # by default. context = SSLContext(protocol) - # SSLv2 considered harmful. - context.options |= OP_NO_SSLv2 - # SSLv3 has problematic security and is only required for really old - # clients such as IE6 on Windows XP - context.options |= OP_NO_SSLv3 if cert_reqs is not None: context.verify_mode = cert_reqs @@ -630,8 +612,8 @@ class SSLSocket(socket): self._sslobj.context = ctx def dup(self): - raise NotImplemented("Can't dup() %s instances" % - self.__class__.__name__) + raise NotImplementedError("Can't dup() %s instances" % + self.__class__.__name__) def _checkClosed(self, msg=None): # raise an exception here if you wish to check for spurious closes diff --git a/Lib/test/allsans.pem b/Lib/test/allsans.pem index 3ee4f59..6eebde7 100644 --- a/Lib/test/allsans.pem +++ b/Lib/test/allsans.pem @@ -1,37 +1,81 @@ -----BEGIN PRIVATE KEY----- -MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAOoy7/QOtTjQ0niE -6uDcTwtkC0R2Tvy1AjVnXohCntZfdzbTGDoYTgXSOLsP8A697jUiJ8VCePGH50xG -Z4DKnAF3a9O3a9nr2pLXb0iY3XOMv+YEBii7CfI+3oxFYgCl0sMgHzDD2ZTVYAsm -DWgLUVsE2gHEccRwrM2tPf2EgR+FAgMBAAECgYEA3qyfyYVSeTrTYxO93x6ZaVMu -A2IZp9zSxMQL9bKiI2GRj+cV2ebSCGbg2btFnD6qBor7FWsmYz+8g6FNN/9sY4az -61rMqMtQvLBe+7L8w70FeTze4qQ4Y1oQri0qD6tBWhDVlpnbI5Py9bkZKD67yVUk -elcEA/5x4PrYXkuqsAECQQD80NjT0mDvaY0JOOaQFSEpMv6QiUA8GGX8Xli7IoKb -tAolPG8rQBa+qSpcWfDMTrWw/aWHuMEEQoP/bVDH9W4FAkEA7SYQbBAKnojZ5A3G -kOHdV7aeivRQxQk/JN8Fb8oKB9Csvpv/BsuGxPKXHdhFa6CBTTsNRtHQw/szPo4l -xMIjgQJAPoMxqibR+0EBM6+TKzteSL6oPXsCnBl4Vk/J5vPgkbmR7KUl4+7j8N8J -b2554TrxKEN/w7CGYZRE6UrRd7ATNQJAWD7Yz41sli+wfPdPU2xo1BHljyl4wMk/ -EPZYbI/PCbdyAH/F935WyQTIjNeEhZc1Zkq6FwdOWw8ns3hrv3rKgQJAHXv1BqUa -czGPIFxX2TNoqtcl6/En4vrxVB1wzsfzkkDAg98kBl7qsF+S3qujSzKikjeaVbI2 -/CyWR2P3yLtOmA== +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQCg/pM6dP7BTFNc +qe6wIJIBB7HjwL42bp0vjcCVl4Z3MRWFswYpfxy+o+8+PguMp4K6zndA5fwNkK/H +3HmtanncUfPqnV0usN0NHQGh/f9xRoNmB1q2L7kTuO99o0KLQgvonRT2snf8rq9n +tPRzhHUGYhog7zzNxetYV309PHpPr19BcKepDtM5RMk2aBnoN5vtItorjXiDosFm +6o5wQHrcupcVydszba6P75BEbc1XIWvq2Fv8muaw4pCe81QYINyLqgcPNO/nF3Os +5EI4HKjCNRSCOhOcWqYctXLXN9lBdMBBvQc3zDmYzh1eIZewzZXPVEQT33xPkhxz +HNmhcIctpWX4LTRF6FulkcbeuZDga3gkZYJf/M6IpU1WYXr6q8sNxbgmRRX/NuHo +V9oDwBzLG07rKUiqRHfjGqoCRmmVeVYpryvXUNjHGH0nlVzz/8lTUxAnJorO3Fdc +I+6zKLUPICdAlvz51AH6yopgPFhrdgA0pVzPO6L5G8SRQCxKhAUCAwEAAQKCAYAa +2jtOTcNMFGH3G7TfFZ+kolbuaPCQ/aQkEV2k1dAswzgWw8RsWXI+7fLyi8C7Zhks +9VD4tyNyU8at7D0zSoYm1Fh9sl+fcQp9rG/gSBA6IYu7EdD0gEM7YeY4K2nm9k4s +Lz8W4q+WqsBA6PK47cfjF6vKAH1AyRk28+jEtPiln9egf5zHWtyqOanh9D0V+Wh9 +hgmjqAYI1rWxZ7/4Qxj7Bfg7Px7blhi+kzOZ5kKQnNd2JT46hM+jgzah/G3zVE+R +FFW6ksmJgZ+dCuSbE7HEJmKms1CWq/1Cll0A3uy4JTDZOrK4KcZQ9UjjWJWvlXQm +uNXSSAp1k287DLVUm9c22SDeXpb9PyKmzyvJvVmMqqBx6QzHZ/L7WPzpUWAoLcU+ +ZHT7vggDymkIO+fcRbUzv8s5R7RnLbcBga51/5OCUvAWDoJXNw0qwYZOIbfTnQgs +8xbCmbMzllyYM/dK3GxQAwfn8Hzk+DbS/NObMjHLCWLfYeUvutXJSNly6Ny+ZcEC +gcEAzo5Y1UFOfBX4MZLIZ69LfgaXj9URobMwqlEwKil8pWQMa951ga3moLt91nOe +SAQz3meFTBX/VAb2ZHLeIf3FoNkiIx48PkxsR/hhLHpvl26zEg3yXs3tv0IFBx2R +EEnLNpQaAQFR9S1yDOaG2rsb17ZDKyp9isDpAENHAmEnT/XJn+Dc0SOH1EVDjUeM +JqToAF/fjIx/RF4oUJCAgOPBMlRy5ywLQk8uDi6ft0NCzzCi0eCuk1Ty3KzWFGwx +7cYRAoHBAMeIPCzHG3No4JGUFunslVwo5TuC7maO6qYKbq0OyvwWfL4b7gjrMBR9 +d5WyZlp/Vf40O463dg8x8qPNOFWp49f3hxTvvfnt2/m3+CQuDOLfqBbHufZApP1J +U9MubUNnDFHHeJ9l0tg2nhiLw24GHeMARZhA/BimMQPY0OpZPpLVxAUArM2EB7hI +glQpYCtdXhqwl1pl0u3TZ08y3BXYNg9BycdpGRMWSsAwsApJRgNuI/dfDKu0uMYF +/pUhXVPatQKBwGgLpAun3dT7bA3sli5ESo6s22OEPGFrVbQ1OUHDrBnTj742TJKJ ++oY0a2q+ypgUJdx94NM2sWquJybqBaKxpf8j4OI3tLjc3h5SqwAwnE13YZRSmifP +K1cP9mBjMFM4GLjhWUfwVkxeG/kLlhpP7fJ2yNbRjHN8QOH1AavdLGRGts1mA1UF +xMHUMfbUd3Bv2L13ja/KhcD2fPA4GcLS9tpXV5nCwdkg8V4LdkBmDR04rotx1f44 +6Czokt2usmfHQQKBwFkufxbUd2SB/72Rnxw27hse/DY5My0Lu70y9HzNG9TIiEDA +YwgBdp/x5D04W58fQuQ3nFcRkOcBwB2OYBuJr5ibvfiRnyvSMHvQykwBeSj+Jjbo +VinGgvfiimDdY2C48jyrFzLHZBHXd5oo/dRzT3Bicri2cvbhcQ7zHY1hDiK7AL3r +q1DALmMjpXzQcXdwZ9suCrgQwtIhpw8zAEOTO7ZeBT3nr5lkYUy9djFixrRJyjGK +fjNQtzVrAHrPStNr8QKBwQDCC0zhsCnTv4sAJmW7LL6Ayd5rbWhUZ6px1xY0yHMA +hehj+xbaiC6cfVr5Rg0ncvaa8AExu4kXpVsupTyNwvC4NgzLHtfBw6WUdOnd1awE +kSrDtDReBt2wByAcQwttQsrJ1/Pt6zcNJJI4Z9s8G4NTcQWJwUhU20N55JQKR//l +OQJqhq9NVhte/ctDjVwOHs/OhDNvxsAWxdjnf/O2up0os+M2bFkmHuaVW0vQbqTQ +mw7Vbzk2Ff5oT6E3kbC8Ur4= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIIDcjCCAtugAwIBAgIJAN5dc9TOWjB7MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV +MIIHMDCCBZigAwIBAgIJALVVA6v9zJS5MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTYwODA1 -MTAyMTExWhcNMjYwODAzMTAyMTExWjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO +IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTgwODI5 +MTQyMzE3WhcNMjgwODI2MTQyMzE3WjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO Q2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0 -aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB -gQDqMu/0DrU40NJ4hOrg3E8LZAtEdk78tQI1Z16IQp7WX3c20xg6GE4F0ji7D/AO -ve41IifFQnjxh+dMRmeAypwBd2vTt2vZ69qS129ImN1zjL/mBAYouwnyPt6MRWIA -pdLDIB8ww9mU1WALJg1oC1FbBNoBxHHEcKzNrT39hIEfhQIDAQABo4IBODCCATQw -ggEwBgNVHREEggEnMIIBI4IHYWxsc2Fuc6AeBgMqAwSgFwwVc29tZSBvdGhlciBp -ZGVudGlmaWVyoDUGBisGAQUCAqArMCmgEBsOS0VSQkVST1MuUkVBTE2hFTAToAMC -AQGhDDAKGwh1c2VybmFtZYEQdXNlckBleGFtcGxlLm9yZ4IPd3d3LmV4YW1wbGUu -b3JnpGcwZTELMAkGA1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMw -IQYDVQQKDBpQeXRob24gU29mdHdhcmUgRm91bmRhdGlvbjEYMBYGA1UEAwwPZGly -bmFtZSBleGFtcGxlhhdodHRwczovL3d3dy5weXRob24ub3JnL4cEfwAAAYcQAAAA -AAAAAAAAAAAAAAAAAYgEKgMEBTANBgkqhkiG9w0BAQsFAAOBgQAy16h+F+nOmeiT -VWR0fc8F/j6FcadbLseAUaogcC15OGxCl4UYpLV88HBkABOoGCpP155qwWTwOrdG -iYPGJSusf1OnJEbvzFejZf6u078bPd9/ZL4VWLjv+FPGkjd+N+/OaqMvgj8Lu99f -3Y/C4S7YbHxxwff6C6l2Xli+q6gnuQ== +aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIB +igKCAYEAoP6TOnT+wUxTXKnusCCSAQex48C+Nm6dL43AlZeGdzEVhbMGKX8cvqPv +Pj4LjKeCus53QOX8DZCvx9x5rWp53FHz6p1dLrDdDR0Bof3/cUaDZgdati+5E7jv +faNCi0IL6J0U9rJ3/K6vZ7T0c4R1BmIaIO88zcXrWFd9PTx6T69fQXCnqQ7TOUTJ +NmgZ6Deb7SLaK414g6LBZuqOcEB63LqXFcnbM22uj++QRG3NVyFr6thb/JrmsOKQ +nvNUGCDci6oHDzTv5xdzrORCOByowjUUgjoTnFqmHLVy1zfZQXTAQb0HN8w5mM4d +XiGXsM2Vz1REE998T5IccxzZoXCHLaVl+C00RehbpZHG3rmQ4Gt4JGWCX/zOiKVN +VmF6+qvLDcW4JkUV/zbh6FfaA8AcyxtO6ylIqkR34xqqAkZplXlWKa8r11DYxxh9 +J5Vc8//JU1MQJyaKztxXXCPusyi1DyAnQJb8+dQB+sqKYDxYa3YANKVczzui+RvE +kUAsSoQFAgMBAAGjggLxMIIC7TCCATAGA1UdEQSCAScwggEjggdhbGxzYW5zoB4G +AyoDBKAXDBVzb21lIG90aGVyIGlkZW50aWZpZXKgNQYGKwYBBQICoCswKaAQGw5L +RVJCRVJPUy5SRUFMTaEVMBOgAwIBAaEMMAobCHVzZXJuYW1lgRB1c2VyQGV4YW1w +bGUub3Jngg93d3cuZXhhbXBsZS5vcmekZzBlMQswCQYDVQQGEwJYWTEXMBUGA1UE +BwwOQ2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5dGhvbiBTb2Z0d2FyZSBGb3Vu +ZGF0aW9uMRgwFgYDVQQDDA9kaXJuYW1lIGV4YW1wbGWGF2h0dHBzOi8vd3d3LnB5 +dGhvbi5vcmcvhwR/AAABhxAAAAAAAAAAAAAAAAAAAAABiAQqAwQFMA4GA1UdDwEB +/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/ +BAIwADAdBgNVHQ4EFgQUoLHAHNTWrHkSCUYkhn5NH0S40CAwgY8GA1UdIwSBhzCB +hIAUoLHAHNTWrHkSCUYkhn5NH0S40CChYaRfMF0xCzAJBgNVBAYTAlhZMRcwFQYD +VQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZv +dW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnOCCQC1VQOr/cyUuTCBgwYIKwYBBQUH +AQEEdzB1MDwGCCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0 +L3Rlc3RjYS9weWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2Eu +cHl0aG9udGVzdC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0 +dHA6Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3Js +MA0GCSqGSIb3DQEBCwUAA4IBgQAeKJKycO2DES98gyR2e/GzPYEw87cCS0cEpiiP +3CEUgzfEbF0X89GDKEey4H3Irvosbvt2hEcf2RNpahLUL/fUv53bDmHNmL8qJg5E +UJVMOHvOpSOjqoqeRuSyG0GnnAuUwcxdrZY6UzLdslhuq9F8UjgHr6KSMx56G9uK +LmTy5njMab0in2xL/YRX/0nogK3BHqpUHrfCdEYZkciRxtAa+OPpWn4dcZi+Fpf7 +ZYSgPLNt+djtFDMIAk5Bo+XDaQdW3dhF0w44enrGAOV0xPE+/jOuenNhKBafjuNb +lkeSr45+QZsi1rd18ny8z3uuaGqIAziFgmllZOH2D8giTn6+5jZcCNZCoGKUkPI9 +l/GMWwxg4HQYYlZcsZzTCem9Rb2XcrasAbmhFapMtR+QAwSed5vKE7ZdtQhj74kB +7Q0E7Lkgpp6BaObb2As8/f0K/UlSVSvrYk+i3JT9wK/qqkRGxsTFEF7N9t0rKu8y +4JdQDtZCI552MsFvYW6m+IOYgxg= -----END CERTIFICATE----- diff --git a/Lib/test/bisect.py b/Lib/test/bisect_cmd.py index 1bf32ef..1bf32ef 100755 --- a/Lib/test/bisect.py +++ b/Lib/test/bisect_cmd.py diff --git a/Lib/test/crashers/warnings_del_crasher.py b/Lib/test/crashers/warnings_del_crasher.py new file mode 100644 index 0000000..02e6805 --- /dev/null +++ b/Lib/test/crashers/warnings_del_crasher.py @@ -0,0 +1,29 @@ +""" +Expose a race in the _warnings module, which is the C backend for the +warnings module. The "_warnings" module tries to access attributes of the +"warnings" module (because of the API it has to support), but doing so +during interpreter shutdown is problematic. Specifically, the call to +PyImport_GetModuleDict() in Python/_warnings.c:get_warnings_attr will +abort() if the modules dict has already been cleaned up. + +This crasher is timing-dependent, and more threads (NUM_THREADS) may be +necessary to expose it reliably on different systems. +""" + +import threading +import warnings + +NUM_THREADS = 10 + +class WarnOnDel(object): + def __del__(self): + warnings.warn("oh no something went wrong", UserWarning) + +def do_work(): + while True: + w = WarnOnDel() + +for i in range(NUM_THREADS): + t = threading.Thread(target=do_work) + t.setDaemon(1) + t.start() diff --git a/Lib/test/dh1024.pem b/Lib/test/dh1024.pem deleted file mode 100644 index a391176..0000000 --- a/Lib/test/dh1024.pem +++ /dev/null @@ -1,7 +0,0 @@ ------BEGIN DH PARAMETERS----- -MIGHAoGBAIbzw1s9CT8SV5yv6L7esdAdZYZjPi3qWFs61CYTFFQnf2s/d09NYaJt -rrvJhIzWavqnue71qXCf83/J3nz3FEwUU/L0mGyheVbsSHiI64wUo3u50wK5Igo0 -RNs/LD0irs7m0icZ//hijafTU+JOBiuA8zMI+oZfU7BGuc9XrUprAgEC ------END DH PARAMETERS----- - -Generated with: openssl dhparam -out dh1024.pem 1024 diff --git a/Lib/test/ffdh3072.pem b/Lib/test/ffdh3072.pem new file mode 100644 index 0000000..ad69bac --- /dev/null +++ b/Lib/test/ffdh3072.pem @@ -0,0 +1,41 @@ + DH Parameters: (3072 bit) + prime: + 00:ff:ff:ff:ff:ff:ff:ff:ff:ad:f8:54:58:a2:bb: + 4a:9a:af:dc:56:20:27:3d:3c:f1:d8:b9:c5:83:ce: + 2d:36:95:a9:e1:36:41:14:64:33:fb:cc:93:9d:ce: + 24:9b:3e:f9:7d:2f:e3:63:63:0c:75:d8:f6:81:b2: + 02:ae:c4:61:7a:d3:df:1e:d5:d5:fd:65:61:24:33: + f5:1f:5f:06:6e:d0:85:63:65:55:3d:ed:1a:f3:b5: + 57:13:5e:7f:57:c9:35:98:4f:0c:70:e0:e6:8b:77: + e2:a6:89:da:f3:ef:e8:72:1d:f1:58:a1:36:ad:e7: + 35:30:ac:ca:4f:48:3a:79:7a:bc:0a:b1:82:b3:24: + fb:61:d1:08:a9:4b:b2:c8:e3:fb:b9:6a:da:b7:60: + d7:f4:68:1d:4f:42:a3:de:39:4d:f4:ae:56:ed:e7: + 63:72:bb:19:0b:07:a7:c8:ee:0a:6d:70:9e:02:fc: + e1:cd:f7:e2:ec:c0:34:04:cd:28:34:2f:61:91:72: + fe:9c:e9:85:83:ff:8e:4f:12:32:ee:f2:81:83:c3: + fe:3b:1b:4c:6f:ad:73:3b:b5:fc:bc:2e:c2:20:05: + c5:8e:f1:83:7d:16:83:b2:c6:f3:4a:26:c1:b2:ef: + fa:88:6b:42:38:61:1f:cf:dc:de:35:5b:3b:65:19: + 03:5b:bc:34:f4:de:f9:9c:02:38:61:b4:6f:c9:d6: + e6:c9:07:7a:d9:1d:26:91:f7:f7:ee:59:8c:b0:fa: + c1:86:d9:1c:ae:fe:13:09:85:13:92:70:b4:13:0c: + 93:bc:43:79:44:f4:fd:44:52:e2:d7:4d:d3:64:f2: + e2:1e:71:f5:4b:ff:5c:ae:82:ab:9c:9d:f6:9e:e8: + 6d:2b:c5:22:36:3a:0d:ab:c5:21:97:9b:0d:ea:da: + 1d:bf:9a:42:d5:c4:48:4e:0a:bc:d0:6b:fa:53:dd: + ef:3c:1b:20:ee:3f:d5:9d:7c:25:e4:1d:2b:66:c6: + 2e:37:ff:ff:ff:ff:ff:ff:ff:ff + generator: 2 (0x2) + recommended-private-length: 276 bits +-----BEGIN DH PARAMETERS----- +MIIBjAKCAYEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz ++8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a +87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7 +YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi +7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD +ssbzSibBsu/6iGtCOGEfz9zeNVs7ZRkDW7w09N75nAI4YbRvydbmyQd62R0mkff3 +7lmMsPrBhtkcrv4TCYUTknC0EwyTvEN5RPT9RFLi103TZPLiHnH1S/9croKrnJ32 +nuhtK8UiNjoNq8Uhl5sN6todv5pC1cRITgq80Gv6U93vPBsg7j/VnXwl5B0rZsYu +N///////////AgECAgIBFA== +-----END DH PARAMETERS----- diff --git a/Lib/test/inspect_fodder.py b/Lib/test/inspect_fodder.py index 5c87ae6..548765c 100644 --- a/Lib/test/inspect_fodder.py +++ b/Lib/test/inspect_fodder.py @@ -56,3 +56,9 @@ class ParrotDroppings: class FesteringGob(MalodorousPervert, ParrotDroppings): pass + +currentframe = inspect.currentframe() +try: + raise Exception() +except: + tb = sys.exc_info()[2] diff --git a/Lib/test/keycert.passwd.pem b/Lib/test/keycert.passwd.pem index e905748..cbb3c3b 100644 --- a/Lib/test/keycert.passwd.pem +++ b/Lib/test/keycert.passwd.pem @@ -1,33 +1,68 @@ -----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED -DEK-Info: DES-EDE3-CBC,1A8D9D2A02EC698A +DEK-Info: DES-EDE3-CBC,D134E931C96D9DEC -kJYbfZ8L0sfe9Oty3gw0aloNnY5E8fegRfQLZlNoxTl6jNt0nIwI8kDJ36CZgR9c -u3FDJm/KqrfUoz8vW+qEnWhSG7QPX2wWGPHd4K94Yz/FgrRzZ0DoK7XxXq9gOtVA -AVGQhnz32p+6WhfGsCr9ArXEwRZrTk/FvzEPaU5fHcoSkrNVAGX8IpSVkSDwEDQr -Gv17+cfk99UV1OCza6yKHoFkTtrC+PZU71LomBabivS2Oc4B9hYuSR2hF01wTHP+ -YlWNagZOOVtNz4oKK9x9eNQpmfQXQvPPTfusexKIbKfZrMvJoxcm1gfcZ0H/wK6P -6wmXSG35qMOOztCZNtperjs1wzEBXznyK8QmLcAJBjkfarABJX9vBEzZV0OUKhy+ -noORFwHTllphbmydLhu6ehLUZMHPhzAS5UN7srtpSN81eerDMy0RMUAwA7/PofX1 -94Me85Q8jP0PC9ETdsJcPqLzAPETEYu0ELewKRcrdyWi+tlLFrpE5KT/s5ecbl9l -7B61U4Kfd1PIXc/siINhU3A3bYK+845YyUArUOnKf1kEox7p1RpD7yFqVT04lRTo -cibNKATBusXSuBrp2G6GNuhWEOSafWCKJQAzgCYIp6ZTV2khhMUGppc/2H3CF6cO -zX0KtlPVZC7hLkB6HT8SxYUwF1zqWY7+/XPPdc37MeEZ87Q3UuZwqORLY+Z0hpgt -L5JXBCoklZhCAaN2GqwFLXtGiRSRFGY7xXIhbDTlE65Wv1WGGgDLMKGE1gOz3yAo -2jjG1+yAHJUdE69XTFHSqSkvaloA1W03LdMXZ9VuQJ/ySXCie6ABAQ== +nuGFEej7vIjkYWSMz5OJeVTNntDRQi6ZM4DBm3g8T7i/0odr3WFqGMMKZcIhLYQf +rgRq7RSKtrJ1y5taVucMV+EuCjyfzDo0TsYt+ZrXv/D08eZhjRmkhoHnGVF0TqQm +nQEXM/ERT4J2RM78dnG+homMkI76qOqxgGbRqQqJo6AiVRcAZ45y8s96bru2TAB8 ++pWjO/v0Je7AFVdwSU52N8OOY6uoSAygW+0UY1WVxbVGJF2XfRsNpPX+YQHYl6e+ +3xM5XBVCgr6kmdAyub5qUJ38X3TpdVGoR0i+CVS9GTr2pSRib1zURAeeHnlqiUZM +4m0Gn9s72nJevU1wxED8pwOhR8fnHEmMKGD2HPhKoOCbzDhwwBZO27TNa1uWeM3f +M5oixKDi2PqMn3y2cDx1NjJtP661688EcJ5a2Ih9BgO9xpnhSyzBWEKcAn0tJB0H +/56M0FW6cdOOIzMveGGL7sHW5E+iOdI1n5e7C6KJUzew78Y9qJnhS53EdI6qTz9R +wsIsj1i070Fk6RbPo6zpLlF6w7Zj8GlZaZA7OZZv9wo5VEV/0ST8gmiiBOBc4C6Y +u9hyLIIu4dFEBKyQHRvBnQSLNpKx6or1OGFDVBay2In9Yh2BHh1+vOj/OIz/wq48 +EHOIV27fRJxLu4jeK5LIGDhuPnMJ8AJYQ0bQOUP6fd7p+TxWkAQZPB/Dx/cs3hxr +nFEdzx+eO+IAsObx/b1EGZyEJyETBslu4GwYX7/KK3HsJhDJ1bdZ//28jOCaoir6 +ZOMT72GRwmVoQTJ0XpccfjHfKJDRLT7C1xvzo4Eibth0hpTZkA75IUYUp6qK/PuJ +kH/qdiC7QIkRKtsrawW4vEDna3YtxIYhQqz9+KwO6u/0gzooZtv1RU4U3ifMDB5u +5P5GAzACRqlY8QYBkM869lvWqzQPHvybC4ak9Yx6/heMO9ddjdIW9BaK8BLxvN/6 +UCD936Y4fWltt09jHZIoxWFykouBwmd7bXooNYXmDRNmjTdVhKJuOEOQw8hDzx7e +pWFJ9Z/V4Qm1tvXbCD7QFqMCDoY3qFvVG8DBqXpmxe1yPfz21FWrT7IuqDXAD3ns +vxfN/2a+Cy04U9FBNVCvWqWIs5AgNpdCMJC2FlXKTy+H3/7rIjNyFyvbX0vxIXtK +liOVNXiyVM++KZXqktqMUDlsJENmIHV9B046luqbgW018fHkyEYlL3iRZGbYegwr +XO9VVIKVPw1BEvJ8VNdGFGuZGepd8qX2ezfYADrNR+4t85HDm8inbjTobSjWuljs +ftUNkOeCHqAvWCFQTLCfdykvV08EJfVY79y7yFPtfRV2gxYokXFifjo3su9sVQr1 +UiIS5ZAsIC1hBXWeXoBN7QVTkFi7Yto6E1q2k10LiT3obpUUUQ/oclhrJOCJVjrS +oRcj2QBy8OT4T9slJr5maTWdgd7Lt6+I6cGQXPaDvjGOJl0eBYM14vhx4rRQWytJ +k07hhHFO4+9CGCuHS8AAy2gR6acYFWt2ZiiNZ0z/iPIHNK4YEyy9aLf6uZH/KQjE +jmHToo7XD6QvCAEC5qTHby3o3LfHIhyZi/4L+AhS4FKUHF6M0peeyYt4z3HaK2d2 +N6mHLPdjwNjra7GOmcns4gzcrdfoF+R293KpPal4PjknvR3dZL4kKP/ougTAM5zv +qDIvRbkHzjP8ChTpoLcJsNVXykNcNkjcSi0GHtIpYjh6QX6P2uvR/S4+Bbb9p9rn +hIy/ovu9tWN2hiPxGPe6torF6BulAxsTYlDercC204AyzsrdA0pr6HBgJH9C6ML1 +TchwodbFJqn9rSv91i1liusAGoOvE81AGBdrXY7LxfSNhYY1IK6yR/POJPTd53sA +uX2/j6Rtoksd/2BHPM6AUnI/2B9slhuzWX2aCtWLeuwvXDS6rYuTigaQmLkzTRfM +dlMI3s9KLXxgi5YVumUZleJWXwBNP7KiKajd+VTSD+7WAhyhM5FIG5wVOaxmy4G2 +TyqZ/Ax9d2VEjTQHWvQlLPQ4Mp0EIz0aEl94K/S8CK8bJRH6+PRkar+dJi1xqlL+ +BYb42At9mEJ8odLlFikvNi1+t7jqXk5jRi5C0xFKx3nTtzoH2zNUeuA3R6vSocVK +45jnze9IkKmxMlJ4loR5sgszdpDCD3kXqjtCcbMTmcrGyzJek3HSOTpiEORoTFOe +Rhg6jH5lm+QcC263oipojS0qEQcnsWJP2CylNYMYHR9O/9NQxT3o2lsRHqZTMELV +uQa/SFH+paQNbZOj8MRwPSqqiIxJFuLswKte1R+W7LKn1yBSM7Pp39lNbzGvJD2E +YRfnCwFpJ54voVAuQ4jXJvigCW2qeCjXlxeD6K2j4eGJEEOmIjIW1wjubyBY6OI3 -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV -BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw -MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH -Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw -gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7 -6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt -pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw -FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd -BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G -lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1 -CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/keycert.pem b/Lib/test/keycert.pem index 64318aa..0d39863 100644 --- a/Lib/test/keycert.pem +++ b/Lib/test/keycert.pem @@ -1,31 +1,66 @@ -----BEGIN PRIVATE KEY----- -MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANtb0+YrKuxevGpm -LrjaUhZSgz6zFAmuGFmKmUbdjmfv9zSmmdsQIksK++jK0Be9LeZy20j6ahOfuVa0 -ufEmPoP7Fy4hXegKZR9cCWcIe/A6H2xWF1IIJLRTLaU8ol/I7T+um5HD5AwAwNPP -USNU0Eegmvp+xxWu3NX2m1Veot85AgMBAAECgYA3ZdZ673X0oexFlq7AAmrutkHt -CL7LvwrpOiaBjhyTxTeSNWzvtQBkIU8DOI0bIazA4UreAFffwtvEuPmonDb3F+Iq -SMAu42XcGyVZEl+gHlTPU9XRX7nTOXVt+MlRRRxL6t9GkGfUAXI3XxJDXW3c0vBK -UL9xqD8cORXOfE06rQJBAP8mEX1ERkR64Ptsoe4281vjTlNfIbs7NMPkUnrn9N/Y -BLhjNIfQ3HFZG8BTMLfX7kCS9D593DW5tV4Z9BP/c6cCQQDcFzCcVArNh2JSywOQ -ZfTfRbJg/Z5Lt9Fkngv1meeGNPgIMLN8Sg679pAOOWmzdMO3V706rNPzSVMME7E5 -oPIfAkEA8pDddarP5tCvTTgUpmTFbakm0KoTZm2+FzHcnA4jRh+XNTjTOv98Y6Ik -eO5d1ZnKXseWvkZncQgxfdnMqqpj5wJAcNq/RVne1DbYlwWchT2Si65MYmmJ8t+F -0mcsULqjOnEMwf5e+ptq5LzwbyrHZYq5FNk7ocufPv/ZQrcSSC+cFwJBAKvOJByS -x56qyGeZLOQlWS2JS3KJo59XuLFGqcbgN9Om9xFa41Yb4N9NvplFivsvZdw3m1Q/ -SPIXQuT8RMPDVNQ= +MIIG/wIBADANBgkqhkiG9w0BAQEFAASCBukwggblAgEAAoIBgQCylKlLaKU+hOvJ +DfriTRLd+IthG5hv28I3A/CGjLICT0rDDtgaXd0uqloJAnjsgn5gMAcStpDW8Rm+ +t6LsrBL+5fBgkyU1r94Rvx0HHoyaZwBBouitVHw28hP3W+smddkqB1UxpGnTeL2B +gj3dVo/WTtRfO+0h0PKw1l98YE1pMTdqIwcOOE/ER0g4hvA/wrxuLhMvlVLMy/lL +58uctqaDUqryNyeerKbVkq4fJyCG5D2TwXVJ3i2DDh0xSt2Y10poZV4M4k8Su9Z5 +8zN2PSvYMT50aqF277v8BaOeYUApBE4kZGIJpo13ATGdEwpUFZ0Fri4zLYUZ1hWb +OC35sKo7OxWQ/+tefNUdgWHob6Vmy777jiYcLwxc3sS9rF3AJe0rMW83kCkR6hmy +A3250E137N/1QumHuT/Nj9rnI/lwt9jfaYkZjoAgT/C97m/mM83cYpGTdoGV1xNo +7G90MhP0di5FnVsrIaSnvkbGT9UgUWx0oVMjocifdG2qIhMI9psCAwEAAQKCAYBT +sHmaPmNaZj59jZCqp0YVQlpHWwBYQ5vD3pPE6oCttm0p9nXt/VkfenQRTthOtmT1 +POzDp00/feP7zeGLmqSYUjgRekPw4gdnN7Ip2PY5kdW77NWwDSzdLxuOS8Rq1MW9 +/Yu+ZPe3RBlDbT8C0IM+Atlh/BqIQ3zIxN4g0pzUlF0M33d6AYfYSzOcUhibOO7H +j84r+YXBNkIRgYKZYbutRXuZYaGuqejRpBj3voVu0d3Ntdb6lCWuClpB9HzfGN0c +RTv8g6UYO4sK3qyFn90ibIR/1GB9watvtoWVZqggiWeBzSWVWRsGEf9O+Cx4oJw1 +IphglhmhbgNksbj7bD24on/icldSOiVkoUemUOFmHWhCm4PnB1GmbD8YMfEdSbks +qDr1Ps1zg4mGOinVD/4cY7vuPFO/HCH07wfeaUGzRt4g0/yLr+XjVofOA3oowyxv +JAzr+niHA3lg5ecj4r7M68efwzN1OCyjMrVJw2RAzwvGxE+rm5NiT08SWlKQZnkC +gcEA4wvyLpIur/UB84nV3XVJ89UMNBLm++aTFzld047BLJtMaOhvNqx6Cl5c8VuW +l261KHjiVzpfNM3/A2LBQJcYkhX7avkqEXlj57cl+dCWAVwUzKmLJTPjfaTTZnYJ +xeN3dMYjJz2z2WtgvfvDoJLukVwIMmhTY8wtqqYyQBJ/l06pBsfw5TNvmVIOQHds +8ASOiFt+WRLk2bl9xrGGayqt3VV93KVRzF27cpjOgEcG74F3c0ZW9snERN7vIYwB +JfrlAoHBAMlahPwMP2TYylG8OzHe7EiehTekSO26LGh0Cq3wTGXYsK/q8hQCzL14 +kWW638vpwXL6L9ntvrd7hjzWRO3vX/VxnYEA6f0bpqHq1tZi6lzix5CTUN5McpDg +QnjenSJNrNjS1zEF8WeY9iLEuDI/M/iUW4y9R6s3WpgQhPDXpSvd2g3gMGRUYhxQ +Xna8auiJeYFq0oNaOxvJj+VeOfJ3ZMJttd+Y7gTOYZcbg3SdRb/kdxYki0RMD2hF +4ZvjJ6CTfwKBwQDiMqiZFTJGQwYqp4vWEmAW+I4r4xkUpWatoI2Fk5eI5T9+1PLX +uYXsho56NxEU1UrOg4Cb/p+TcBc8PErkGqR0BkpxDMOInTOXSrQe6lxIBoECVXc3 +HTbrmiay0a5y5GfCgxPKqIJhfcToAceoVjovv0y7S4yoxGZKuUEe7E8JY2iqRNAO +yOvKCCICv/hcN235E44RF+2/rDlOltagNej5tY6rIFkaDdgOF4bD7f9O5eEni1Bg +litfoesDtQP/3rECgcEAkQfvQ7D6tIPmbqsbJBfCr6fmoqZllT4FIJN84b50+OL0 +mTGsfjdqC4tdhx3sdu7/VPbaIqm5NmX10bowWgWSY7MbVME4yQPyqSwC5NbIonEC +d6N0mzoLR0kQ+Ai4u+2g82gicgAq2oj1uSNi3WZi48jQjHYFulCbo246o1NgeFFK +77WshYe2R1ioQfQDOU1URKCR0uTaMHClgfu112yiGd12JAD+aF3TM0kxDXz+sXI5 +SKy311DFxECZeXRLpcC3AoHBAJkNMJWTyPYbeVu+CTQkec8Uun233EkXa2kUNZc/ +5DuXDaK+A3DMgYRufTKSPpDHGaCZ1SYPInX1Uoe2dgVjWssRL2uitR4ENabDoAOA +ICVYXYYNagqQu5wwirF0QeaMXo1fjhuuHQh8GsMdXZvYEaAITZ9/NG5x/oY08+8H +kr78SMBOPy3XQn964uKG+e3JwpOG14GKABdAlrHKFXNWchu/6dgcYXB87mrC/GhO +zNwzC+QhFTZoOomFoqMgFWujng== -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV -BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw -MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH -Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw -gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7 -6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt -pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw -FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd -BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G -lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1 -CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/keycert2.pem b/Lib/test/keycert2.pem index c4a18bf..ed6ae85 100644 --- a/Lib/test/keycert2.pem +++ b/Lib/test/keycert2.pem @@ -1,31 +1,66 @@ -----BEGIN PRIVATE KEY----- -MIICdQIBADANBgkqhkiG9w0BAQEFAASCAl8wggJbAgEAAoGBANcLaMB7T/Wi9DBc -PltGzgt8cxsv55m7PQPHMZvn6Ke8xmNqcmEzib8opRwKGrCV6TltKeFlNSg8dwQK -Tl4ktyTkGCVweRQJ37AkBayvEBml5s+QD4vlhqkJPsL/Nsd+fnqngOGc5+59+C6r -s3XpiLlF5ah/z8q92Mnw54nypw1JAgMBAAECgYBE3t2Mj7GbDLZB6rj5yKJioVfI -BD6bSJEQ7bGgqdQkLFwpKMU7BiN+ekjuwvmrRkesYZ7BFgXBPiQrwhU5J28Tpj5B -EOMYSIOHfzdalhxDGM1q2oK9LDFiCotTaSdEzMYadel5rmKXJ0zcK2Jho0PCuECf -tf/ghRxK+h1Hm0tKgQJBAO6MdGDSmGKYX6/5kPDje7we/lSLorSDkYmV0tmVShsc -JxgaGaapazceA/sHL3Myx7Eenkip+yPYDXEDFvAKNDECQQDmxsT9NOp6mo7ISvky -GFr2vVHsJ745BMWoma4rFjPBVnS8RkgK+b2EpDCdZSrQ9zw2r8sKTgrEyrDiGTEg -wJyZAkA8OOc0flYMJg2aHnYR6kwVjPmGHI5h5gk648EMPx0rROs1sXkiUwkHLCOz -HvhCq+Iv+9vX2lnVjbiu/CmxRdIxAkA1YEfzoKeTD+hyXxTgB04Sv5sRGegfXAEz -i8gC4zG5R/vcCA1lrHmvEiLEZL/QcT6WD3bQvVg0SAU9ZkI8pxARAkA7yqMSvP1l -gJXy44R+rzpLYb1/PtiLkIkaKG3x9TUfPnfD2jY09fPkZlfsRU3/uS09IkhSwimV -d5rWoljEfdou +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQDKjrjWZlfOs1Ch +qt1RoyLfqyXbHVXIAW0fTzAxfJnxvFOiWqAAKgC2qVQM8Y080kRUuRaXP/w9ywXT ++MzX6tByy5VbTYJYyTjHOH46EWLNdcqEJs4+FCVqOIYrQPQ6pGAhCXmgBy4Vb42J +ABLwb+Kt+y2Dk15tggVcAHP2Khri+lRXWvda+kZAe2F1IojmuWyCTy3FEYHic5qN +BsXcf6u1oyFV8MybOuz1zGj3vd2C+dEKO4Ohw9rRwnvHSatjM+CfwiXf8kTXzDBF +Z/8W3+6yA49pHxRbG7FE3K1TAPhkrp+BVTIUOcdI74wEA6UEkWFF5sQcmmAth59M +EQrl2CXorftEPhsKZE59dUP1+nYAPvR/mTySNCSw7/rvdf+csRSZ5ollMu/lxsht +ulJYJI03+IiDTn47FI5D+IF25REK7d4LzGIo6T73ktsT+qpSXHuTWC+IABm8AMF9 +7ljxHSwMRU/z+O5uiONRItDAgKH/OItFG54PtY2vAhaO0YiZrZcCAwEAAQKCAYB2 +hTo8IVghlySH5B1p5kXCkDcvVaPaypLaLhCp9Blzq9lX9yUF043lU4Ddrf0RaIsY +88/3IjZqxb+cP0lE0Z20fdDfwqORZfQ2BaU+PuwMAm9EEhy9kDYwR/ChoHkHUyT4 +T7392BWr70Dmt8ddLmp5mK4R/gnTk6+lHJK9p/dhdk4haxWvAyBWHJty2Yk3T6nh +OYkzdUIFidUVza+6jG2hc1lPGv3tmnYKgNeulkblm10oWphz79C6ycx5WG7TNgef +CQ3z7//Nn89YTiaUBjLvoLvxRTMwO96r7E/FaslSl/fWnF3HP3lut26Z/mNfhiwj +qn7AhUwpSNPV0qcxFWXr/rXUjdk745wv8wOODK8atjjE/vt/MRBK0rAOIPSm3ecx +37PKNtR4i+sNeDEcY1IyTHE6wFvJSy5y8AFpn5y8tbqYfhlEVWZ4pcnlrKxhWm7j +oBkB/4GBjKQgbQ7ttym9eNG1wIbZ8v9N06+yeLs/NCc4bFZEgcWjFqBH1bLtAYEC +gcEA8tt8iYNqbsDH2ognjEmbbBxrDBmyYcEKRpg1i1SUopcZl8i93IHpG7EgJIaj +l7aWSbASAxjnK02t0VZ3nNS60acibzRwY/+e8OrSqlQdMXlAB2ggBA86drDJpfBl +WGJG8TJVY9bc1TU2uuwtZR1LAMSsRHVp+3IvKLpHrne5exPd3x6KGYcuaM+Uk/rE +u6tLsFNwaCdh+iBFFDT2bnYIw7jAsokJUkwxMVxSC0/21s2blhO/q5LsN1gFC1kN +TbpXAoHBANWE7TmG2szPvujPwrK18v6iJlHCA2n50AgAQXrsetj2JcF3HYHYdHnq +z36MQ6FpBKOiQumozWvb32WTjEwdG2kix7GEfam4DAUBdqYuCHzPcR12K5Tc8hsX +NG7JXUAeS8ZJEiOdu95X59JHyBxUQtNfte5rcbaV17SVw6K6bsWVJnj60YjtJrpa +xHvv1ZRnT2WEzJGpA+ii1h3I52N7ipGBiw172qcW+bKJukMi8eHxx5CC9e5tBpnu +C+Ou/eYewQKBwHxNa0jXQrq9YY2w8s0TP8HuKbxfyrXOIHxRm9ZczFcMD8VosgUT +WUUbO+B2KXWVtwawYAfFz0ySzcy//SkAmT6F1VIl/QCx7aBSENGti+Ous98WpIxv +XvUxN4T/rl+2raj2ok4fw5g9TG4QRIvkmmciQyonDr/sicbG0bmy/fTJDl8NOpIm +ZtKurNWxHNERtAPkMTyeK7/ilHjrQtb3AzVqcvbuvR6qcONa5YN0wlrfkisWoJwo +707EdpCAXBbUsQKBwQCnpzcpu2Sj+t9ZKIElF87T93gFLETH+ppJHgJMRdDz+NqO +fTwTD2XtsNz57aLQ44f8AFVv6NZbQYq41FEOFrDGLcQE9BZDpDrz10FVnMGXVr7n +tjjkK1SCxwapkr0AsoknCYsPojO4kud46loLPHI4TGeq7HyeNCvqJMo3RRHjXIiX +58GNNUD6hHjRI/FdFH14Jf0GxmJGUU20l2Jwb7nPJJuNm9mE53pqoNA7FP4+Pj1H +kD0Q2FSdmxeE0IuWHEECgcBgw6ogJ/FRRGLcym+aApqP9BChK+W8FDfDc9Mi4p/J +g+XmetWNFGCGTlOefGqUDIkwSG+QVOEN3hxziXbsjnvfpGApqoaulAI5oRvrwIcj +QIvD2mt0PB52k5ZL9QL2K9sgBa43BJDyCKooMAlTy2XMM+NyXVxQKmzf3r3jQ5sl +Rptk7ro38a9G8Rs99RFDyOmP1haOM0KXZvPksN4nsXuTlE01cnwnI29XKAlEZaoA +pQPLXD8W/KK4mwDbmokYXmo= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIICXTCCAcagAwIBAgIJALVQzebTtrXFMA0GCSqGSIb3DQEBBQUAMGIxCzAJBgNV +MIIEYjCCAsqgAwIBAgIJAJm2YulYpr+6MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u IFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTAeFw0x -NDExMjMxNzAwMDdaFw0yNDExMjAxNzAwMDdaMGIxCzAJBgNVBAYTAlhZMRcwFQYD +ODA4MjkxNDIzMTZaFw0yODA4MjYxNDIzMTZaMGIxCzAJBgNVBAYTAlhZMRcwFQYD VQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZv -dW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTCBnzANBgkqhkiG9w0BAQEF -AAOBjQAwgYkCgYEA1wtowHtP9aL0MFw+W0bOC3xzGy/nmbs9A8cxm+fop7zGY2py -YTOJvyilHAoasJXpOW0p4WU1KDx3BApOXiS3JOQYJXB5FAnfsCQFrK8QGaXmz5AP -i+WGqQk+wv82x35+eqeA4Zzn7n34LquzdemIuUXlqH/Pyr3YyfDnifKnDUkCAwEA -AaMbMBkwFwYDVR0RBBAwDoIMZmFrZWhvc3RuYW1lMA0GCSqGSIb3DQEBBQUAA4GB -AKuay3vDKfWzt5+ch/HHBsert84ISot4fUjzXDA/oOgTOEjVcSShHxqNShMOW1oA -QYBpBB/5Kx5RkD/w6imhucxt2WQPRgjX4x4bwMipVH/HvFDp03mG51/Cpi1TyZ74 -El7qa/Pd4lHhOLzMKBA6503fpeYSFUIBxZbGLqylqRK7 +dW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTCCAaIwDQYJKoZIhvcNAQEB +BQADggGPADCCAYoCggGBAMqOuNZmV86zUKGq3VGjIt+rJdsdVcgBbR9PMDF8mfG8 +U6JaoAAqALapVAzxjTzSRFS5Fpc//D3LBdP4zNfq0HLLlVtNgljJOMc4fjoRYs11 +yoQmzj4UJWo4hitA9DqkYCEJeaAHLhVvjYkAEvBv4q37LYOTXm2CBVwAc/YqGuL6 +VFda91r6RkB7YXUiiOa5bIJPLcURgeJzmo0Gxdx/q7WjIVXwzJs67PXMaPe93YL5 +0Qo7g6HD2tHCe8dJq2Mz4J/CJd/yRNfMMEVn/xbf7rIDj2kfFFsbsUTcrVMA+GSu +n4FVMhQ5x0jvjAQDpQSRYUXmxByaYC2Hn0wRCuXYJeit+0Q+GwpkTn11Q/X6dgA+ +9H+ZPJI0JLDv+u91/5yxFJnmiWUy7+XGyG26UlgkjTf4iINOfjsUjkP4gXblEQrt +3gvMYijpPveS2xP6qlJce5NYL4gAGbwAwX3uWPEdLAxFT/P47m6I41Ei0MCAof84 +i0Ubng+1ja8CFo7RiJmtlwIDAQABoxswGTAXBgNVHREEEDAOggxmYWtlaG9zdG5h +bWUwDQYJKoZIhvcNAQELBQADggGBAMIVLp6e6saH2NQSg8iFg8Ewg/K/etI++jHo +gCJ697AY02wtfrBox1XtljlmI2xpJtVAYZWHhrNqwrEG43aB7YEV6RqTcG6QUVqa +NbD8iNCnMKm7fP89hZizmqA1l4aHnieI3ucOqpgooM7FQwLX6qk+rSue6lD5N/5f +avsublnj8rNKyDfHpQ3AWduLoj8QqctpzI3CqoDZNLNzaDnzVWpxT1SKDQ88q7VI +W5zb+lndpdQlCu3v5HM4w5UpwL/k1htl/z6PnPseS2UdlXv6A8KITnCLg5PLP4tz +2oTAg9gjOtRP/0uwkhvicwoFzFJNVT813lzTLE1jlobMPiZhsS1mjaJGPD9GQZDK +ny3j8ogrIRGjnI4xpOMNNDVphcvwtV8fRbvURSHCj9Y4kCLpD5ODuoyEyLYicJIv +GZP456GP0iSCK5GKO0ij/YzGCkPWD5zA+mYFpMMGZPTwajenMw7TVaPXcc9CZBtr +oOjwwiLEqdkpxUj13mJYTlt5wsS/Kw== -----END CERTIFICATE----- diff --git a/Lib/test/keycert3.pem b/Lib/test/keycert3.pem index 5bfa62c..e0a8205 100644 --- a/Lib/test/keycert3.pem +++ b/Lib/test/keycert3.pem @@ -1,73 +1,164 @@ -----BEGIN PRIVATE KEY----- -MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAMLgD0kAKDb5cFyP -jbwNfR5CtewdXC+kMXAWD8DLxiTTvhMW7qVnlwOm36mZlszHKvsRf05lT4pegiFM -9z2j1OlaN+ci/X7NU22TNN6crYSiN77FjYJP464j876ndSxyD+rzys386T+1r1aZ -aggEdkj1TsSsv1zWIYKlPIjlvhuxAgMBAAECgYA0aH+T2Vf3WOPv8KdkcJg6gCRe -yJKXOWgWRcicx/CUzOEsTxmFIDPLxqAWA3k7v0B+3vjGw5Y9lycV/5XqXNoQI14j -y09iNsumds13u5AKkGdTJnZhQ7UKdoVHfuP44ZdOv/rJ5/VD6F4zWywpe90pcbK+ -AWDVtusgGQBSieEl1QJBAOyVrUG5l2yoUBtd2zr/kiGm/DYyXlIthQO/A3/LngDW -5/ydGxVsT7lAVOgCsoT+0L4efTh90PjzW8LPQrPBWVMCQQDS3h/FtYYd5lfz+FNL -9CEe1F1w9l8P749uNUD0g317zv1tatIqVCsQWHfVHNdVvfQ+vSFw38OORO00Xqs9 -1GJrAkBkoXXEkxCZoy4PteheO/8IWWLGGr6L7di6MzFl1lIqwT6D8L9oaV2vynFT -DnKop0pa09Unhjyw57KMNmSE2SUJAkEArloTEzpgRmCq4IK2/NpCeGdHS5uqRlbh -1VIa/xGps7EWQl5Mn8swQDel/YP3WGHTjfx7pgSegQfkyaRtGpZ9OQJAa9Vumj8m -JAAtI0Bnga8hgQx7BhTQY4CadDxyiRGOGYhwUzYVCqkb2sbVRH9HnwUaJT7cWBY3 -RnJdHOMXWem7/w== +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQCfKC83Qe9/ZGMW +YhbpARRiKco6mJI9CNNeaf7A89TE+w5Y3GSwS8uzqp5C6QebZzPNueg8HYoTwN85 +Z3xM036/Qw9KhQVth+XDAqM+19e5KHkYcxg3d3ZI1HgY170eakaLBvMDN5ULoFOw +Is2PtwM2o9cjd5mfSuWttI6+fCqop8/l8cerG9iX2GH39p3iWwWoTZuYndAA9qYv +07YWajuQ1ESWKPjHYGTnMvu4xIzibC1mXd2M6u/IjNO6g426SKFaRDWQkx01gIV/ +CyKs9DgZoeMHkKZuPqZVOxOK+A/NrmrqHFsPIsrs5wk7QAVju5/X1skpn/UGQlmM +RwBaQULOs1FagA+54RXU6qUPW0YmhJ4xOB4gHHD1vjAKEsRZ7/6zcxMyOm+M1DbK +RTH4NWjVWpnY8XaVGdRhtTpH9MjycpKhF+D2Zdy2tQXtqu2GdcMnUedt13fn9xDu +P4PophE0ip/IMgn+kb4m9e+S+K9lldQl0B+4BcGWAqHelh2KuU0CAwEAAQKCAYEA +lKiWIYjmyRjdLKUGPTES9vWNvNmRjozV0RQ0LcoSbMMLDZkeO0UwyWqOVHUQ8+ib +jIcfEjeNJxI57oZopeHOO5vJhpNlFH+g7ltiW2qERqA1K88lSXm99Bzw6FNqhCRE +K8ub5N9fyfJA+P4o/xm0WK8EXk5yIUV17p/9zJJxzgKgv2jsVTi3QG2OZGvn4Oug +ByomMZEGHkBDzdxz8c/cP1Tlk1RFuwSgews178k2xq7AYSM/s0YmHi7b/RSvptX6 +1v8P8kXNUe4AwTaNyrlvF2lwIadZ8h1hA7tCE2n44b7a7KfhAkwcbr1T59ioYh6P +zxsyPT678uD51dbtD/DXJCcoeeFOb8uzkR2KNcrnQzZpCJnRq4Gp5ybxwsxxuzpr +gz0gbNlhuWtE7EoSzmIK9t+WTS7IM2CvZymd6/OAh1Fuw6AQhSp64XRp3OfMMAAC +Ie2EPtKj4islWGT8VoUjuRYGmdRh4duAH1dkiAXOWA3R7y5a1/y/iE8KE8BtxocB +AoHBAM8aiURgpu1Fs0Oqz6izec7KSLL3l8hmW+MKUOfk/Ybng6FrTFsL5YtzR+Ap +wW4wwWnnIKEc1JLiZ7g8agRETK8hr5PwFXUn/GSWC0SMsazLJToySQS5LOV0tLzK +kJ3jtNU7tnlDGNkCHTHSoVL2T/8t+IkZI/h5Z6wjlYPvU2Iu0nVIXtiG+alv4A6M +Hrh9l5or4mjB6rGnVXeYohLkCm6s/W97ahVxLMcEdbsBo1prm2JqGnSoiR/tEFC/ +QHQnbQKBwQDEu7kW0Yg9sZ89QtYtVQ1YpixFZORaUeRIRLnpEs1w7L1mCbOZ2Lj9 +JHxsH05cYAc7HJfPwwxv3+3aGAIC/dfu4VSwEFtatAzUpzlhzKS5+HQCWB4JUNNU +MQ3+FwK2xQX4Ph8t+OzrFiYcK2g0An5UxWMa2HWIAWUOhnTOydAVsoH6yP31cVm4 +0hxoABCwflaNLNGjRUyfBpLTAcNu/YtcE+KREy7YAAgXXrhRSO4XpLsSXwLnLT7/ +YOkoBWDcTWECgcBPWnSUDZCIQ3efithMZJBciqd2Y2X19Dpq8O31HImD4jtOY0V7 +cUB/wSkeHAGwjd/eCyA2e0x8B2IEdqmMfvr+86JJxekC3dJYXCFvH5WIhsH53YCa +3bT1KlWCLP9ib/g+58VQC0R/Cc9T4sfLePNH7D5ZkZd1wlbV30CPr+i8KwKay6MD +xhvtLx+jk07GE+E9wmjbCMo7TclyrLoVEOlqZMAqshgApT+p9eyCPetwXuDHwa3n +WxhHclcZCV7R4rUCgcAkdGSnxcvpIrDPOUNWwxvmAWTStw9ZbTNP8OxCNCm9cyDl +d4bAS1h8D/a+Uk7C70hnu7Sl2w7C7Eu2zhwRUdhhe3+l4GINPK/j99i6NqGPlGpq +xMlMEJ4YS768BqeKFpg0l85PRoEgTsphDeoROSUPsEPdBZ9BxIBlYKTkbKESZDGR +twzYHljx1n1NCDYPflmrb1KpXn4EOcObNghw2KqqNUUWfOeBPwBA1FxzM4BrAStp +DBINpGS4Dc0mjViVegECgcA3hTtm82XdxQXj9LQmb/E3lKx/7H87XIOeNMmvjYuZ +iS9wKrkF+u42vyoDxcKMCnxP5056wpdST4p56r+SBwVTHcc3lGBSGcMTIfwRXrj3 +thOA2our2n4ouNIsYyTlcsQSzifwmpRmVMRPxl9fYVdEWUgB83FgHT0D9avvZnF9 +t9OccnGJXShAIZIBADhVj/JwG4FbaX42NijD5PNpVLk1Y17OV0I576T9SfaQoBjJ +aH1M/zC4aVaS0DYB/Gxq7v8= -----END PRIVATE KEY----- Certificate: Data: - Version: 1 (0x0) - Serial Number: 12723342612721443281 (0xb09264b1f2da21d1) - Signature Algorithm: sha1WithRSAEncryption + Version: 3 (0x2) + Serial Number: + cb:2d:80:99:5a:69:52:5c + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 4 19:47:07 2013 GMT - Not After : Nov 13 19:47:07 2022 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Jul 7 14:23:16 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (1024 bit) + Public-Key: (3072 bit) Modulus: - 00:c2:e0:0f:49:00:28:36:f9:70:5c:8f:8d:bc:0d: - 7d:1e:42:b5:ec:1d:5c:2f:a4:31:70:16:0f:c0:cb: - c6:24:d3:be:13:16:ee:a5:67:97:03:a6:df:a9:99: - 96:cc:c7:2a:fb:11:7f:4e:65:4f:8a:5e:82:21:4c: - f7:3d:a3:d4:e9:5a:37:e7:22:fd:7e:cd:53:6d:93: - 34:de:9c:ad:84:a2:37:be:c5:8d:82:4f:e3:ae:23: - f3:be:a7:75:2c:72:0f:ea:f3:ca:cd:fc:e9:3f:b5: - af:56:99:6a:08:04:76:48:f5:4e:c4:ac:bf:5c:d6: - 21:82:a5:3c:88:e5:be:1b:b1 + 00:9f:28:2f:37:41:ef:7f:64:63:16:62:16:e9:01: + 14:62:29:ca:3a:98:92:3d:08:d3:5e:69:fe:c0:f3: + d4:c4:fb:0e:58:dc:64:b0:4b:cb:b3:aa:9e:42:e9: + 07:9b:67:33:cd:b9:e8:3c:1d:8a:13:c0:df:39:67: + 7c:4c:d3:7e:bf:43:0f:4a:85:05:6d:87:e5:c3:02: + a3:3e:d7:d7:b9:28:79:18:73:18:37:77:76:48:d4: + 78:18:d7:bd:1e:6a:46:8b:06:f3:03:37:95:0b:a0: + 53:b0:22:cd:8f:b7:03:36:a3:d7:23:77:99:9f:4a: + e5:ad:b4:8e:be:7c:2a:a8:a7:cf:e5:f1:c7:ab:1b: + d8:97:d8:61:f7:f6:9d:e2:5b:05:a8:4d:9b:98:9d: + d0:00:f6:a6:2f:d3:b6:16:6a:3b:90:d4:44:96:28: + f8:c7:60:64:e7:32:fb:b8:c4:8c:e2:6c:2d:66:5d: + dd:8c:ea:ef:c8:8c:d3:ba:83:8d:ba:48:a1:5a:44: + 35:90:93:1d:35:80:85:7f:0b:22:ac:f4:38:19:a1: + e3:07:90:a6:6e:3e:a6:55:3b:13:8a:f8:0f:cd:ae: + 6a:ea:1c:5b:0f:22:ca:ec:e7:09:3b:40:05:63:bb: + 9f:d7:d6:c9:29:9f:f5:06:42:59:8c:47:00:5a:41: + 42:ce:b3:51:5a:80:0f:b9:e1:15:d4:ea:a5:0f:5b: + 46:26:84:9e:31:38:1e:20:1c:70:f5:be:30:0a:12: + c4:59:ef:fe:b3:73:13:32:3a:6f:8c:d4:36:ca:45: + 31:f8:35:68:d5:5a:99:d8:f1:76:95:19:d4:61:b5: + 3a:47:f4:c8:f2:72:92:a1:17:e0:f6:65:dc:b6:b5: + 05:ed:aa:ed:86:75:c3:27:51:e7:6d:d7:77:e7:f7: + 10:ee:3f:83:e8:a6:11:34:8a:9f:c8:32:09:fe:91: + be:26:f5:ef:92:f8:af:65:95:d4:25:d0:1f:b8:05: + c1:96:02:a1:de:96:1d:8a:b9:4d Exponent: 65537 (0x10001) - Signature Algorithm: sha1WithRSAEncryption - 2f:42:5f:a3:09:2c:fa:51:88:c7:37:7f:ea:0e:63:f0:a2:9a: - e5:5a:e2:c8:20:f0:3f:60:bc:c8:0f:b6:c6:76:ce:db:83:93: - f5:a3:33:67:01:8e:04:cd:00:9a:73:fd:f3:35:86:fa:d7:13: - e2:46:c6:9d:c0:29:53:d4:a9:90:b8:77:4b:e6:83:76:e4:92: - d6:9c:50:cf:43:d0:c6:01:77:61:9a:de:9b:70:f7:72:cd:59: - 00:31:69:d9:b4:ca:06:9c:6d:c3:c7:80:8c:68:e6:b5:a2:f8: - ef:1d:bb:16:9f:77:77:ef:87:62:22:9b:4d:69:a4:3a:1a:f1: - 21:5e:8c:32:ac:92:fd:15:6b:18:c2:7f:15:0d:98:30:ca:75: - 8f:1a:71:df:da:1d:b2:ef:9a:e8:2d:2e:02:fd:4a:3c:aa:96: - 0b:06:5d:35:b3:3d:24:87:4b:e0:b0:58:60:2f:45:ac:2e:48: - 8a:b0:99:10:65:27:ff:cc:b1:d8:fd:bd:26:6b:b9:0c:05:2a: - f4:45:63:35:51:07:ed:83:85:fe:6f:69:cb:bb:40:a8:ae:b6: - 3b:56:4a:2d:a4:ed:6d:11:2c:4d:ed:17:24:fd:47:bc:d3:41: - a2:d3:06:fe:0c:90:d8:d8:94:26:c4:ff:cc:a1:d8:42:77:eb: - fc:a9:94:71 + X509v3 extensions: + X509v3 Subject Alternative Name: + DNS:localhost + X509v3 Key Usage: critical + Digital Signature, Key Encipherment + X509v3 Extended Key Usage: + TLS Web Server Authentication, TLS Web Client Authentication + X509v3 Basic Constraints: critical + CA:FALSE + X509v3 Subject Key Identifier: + 8F:EA:1D:E3:33:5C:00:16:B3:8B:6F:6B:6F:D3:4C:CB:B5:CB:7C:55 + X509v3 Authority Key Identifier: + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 + DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server + serial:CB:2D:80:99:5A:69:52:5B + + Authority Information Access: + CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer + OCSP - URI:http://testca.pythontest.net/testca/ocsp/ + + X509v3 CRL Distribution Points: + + Full Name: + URI:http://testca.pythontest.net/testca/revocation.crl + + Signature Algorithm: sha256WithRSAEncryption + 27:f5:8c:59:10:f4:c6:e7:28:00:bf:ba:8d:7b:13:03:f1:1c: + a6:5f:b3:06:55:a4:22:b9:db:b2:d5:46:bd:f7:0c:dd:43:6e: + b4:79:65:67:21:0c:2a:55:ee:40:8e:85:9f:9f:47:bb:0a:2a: + 4d:b6:64:74:98:a0:7f:ae:dc:f1:2e:db:42:77:18:e0:75:8b: + 26:35:68:c3:41:ed:6b:c8:77:72:6f:6a:9a:5d:55:69:02:fd: + 5a:54:c8:57:cb:b0:65:03:16:e2:0f:00:39:99:66:a0:9b:88: + 93:17:e2:5a:2d:79:35:5f:97:57:78:c4:af:f5:99:5e:86:ab: + d3:11:ad:1a:a2:0d:fa:52:10:b9:fe:bf:9d:ce:33:d9:86:b2: + 9c:16:f8:d6:75:08:8a:db:0a:e5:b4:2b:16:7f:b4:f9:2a:9f: + c3:d2:77:d7:cd:65:1e:f4:6c:1e:eb:59:b9:f0:ae:5f:a4:1f: + cc:4a:c4:b9:7a:a9:d9:6b:32:68:3b:e1:65:b0:84:b7:90:c4: + ae:fe:f4:37:4f:21:a0:de:9f:3a:b1:e5:cc:16:04:66:3f:0b: + 41:dc:42:3d:20:3e:ec:b7:95:2b:35:57:fa:be:7f:b6:3a:ba: + ca:4f:58:fe:75:3e:08:89:2c:8c:b0:5d:2e:f9:89:10:2b:f9: + 41:46:4f:3c:00:b7:27:d3:65:24:28:17:23:26:31:42:ea:7e: + 4e:93:e4:7b:68:54:ca:9f:46:f3:ef:2b:e9:85:0c:b5:84:b2: + d5:35:34:80:75:2b:f0:91:23:b8:08:01:8e:b9:0a:54:d4:fb: + 34:52:fe:d9:45:f0:80:3b:b6:c1:6f:82:d1:1f:f2:3b:08:f6: + 46:a6:96:27:61:4b:58:32:7a:0e:1d:59:c5:44:ad:5e:1a:79: + 33:c1:d4:05:2f:4a:d3:d8:42:42:8d:33:e3:63:ca:d5:87:97: + 9b:4d:b8:1a:03:34:bb:1c:d2:02:3f:59:23:e2:23:80:88:63: + c2:f0:a2:63:a8:8b -----BEGIN CERTIFICATE----- -MIICpDCCAYwCCQCwkmSx8toh0TANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJY -WTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNV -BAMMDW91ci1jYS1zZXJ2ZXIwHhcNMTMwMTA0MTk0NzA3WhcNMjIxMTEzMTk0NzA3 -WjBfMQswCQYDVQQGEwJYWTEXMBUGA1UEBxMOQ2FzdGxlIEFudGhyYXgxIzAhBgNV -BAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRIwEAYDVQQDEwlsb2NhbGhv -c3QwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMLgD0kAKDb5cFyPjbwNfR5C -tewdXC+kMXAWD8DLxiTTvhMW7qVnlwOm36mZlszHKvsRf05lT4pegiFM9z2j1Ola -N+ci/X7NU22TNN6crYSiN77FjYJP464j876ndSxyD+rzys386T+1r1aZaggEdkj1 -TsSsv1zWIYKlPIjlvhuxAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAC9CX6MJLPpR -iMc3f+oOY/CimuVa4sgg8D9gvMgPtsZ2ztuDk/WjM2cBjgTNAJpz/fM1hvrXE+JG -xp3AKVPUqZC4d0vmg3bkktacUM9D0MYBd2Ga3ptw93LNWQAxadm0ygacbcPHgIxo -5rWi+O8duxafd3fvh2Iim01ppDoa8SFejDKskv0VaxjCfxUNmDDKdY8acd/aHbLv -mugtLgL9SjyqlgsGXTWzPSSHS+CwWGAvRawuSIqwmRBlJ//Msdj9vSZruQwFKvRF -YzVRB+2Dhf5vacu7QKiutjtWSi2k7W0RLE3tFyT9R7zTQaLTBv4MkNjYlCbE/8yh -2EJ36/yplHE= +MIIF8TCCBFmgAwIBAgIJAMstgJlaaVJcMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV +BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA3MDcx +NDIzMTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxv +Y2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAJ8oLzdB739k +YxZiFukBFGIpyjqYkj0I015p/sDz1MT7DljcZLBLy7OqnkLpB5tnM8256DwdihPA +3zlnfEzTfr9DD0qFBW2H5cMCoz7X17koeRhzGDd3dkjUeBjXvR5qRosG8wM3lQug +U7AizY+3Azaj1yN3mZ9K5a20jr58Kqinz+Xxx6sb2JfYYff2neJbBahNm5id0AD2 +pi/TthZqO5DURJYo+MdgZOcy+7jEjOJsLWZd3Yzq78iM07qDjbpIoVpENZCTHTWA +hX8LIqz0OBmh4weQpm4+plU7E4r4D82uauocWw8iyuznCTtABWO7n9fWySmf9QZC +WYxHAFpBQs6zUVqAD7nhFdTqpQ9bRiaEnjE4HiAccPW+MAoSxFnv/rNzEzI6b4zU +NspFMfg1aNVamdjxdpUZ1GG1Okf0yPJykqEX4PZl3La1Be2q7YZ1wydR523Xd+f3 +EO4/g+imETSKn8gyCf6Rvib175L4r2WV1CXQH7gFwZYCod6WHYq5TQIDAQABo4IB +wDCCAbwwFAYDVR0RBA0wC4IJbG9jYWxob3N0MA4GA1UdDwEB/wQEAwIFoDAdBgNV +HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E +FgQUj+od4zNcABazi29rb9NMy7XLfFUwfQYDVR0jBHYwdIAU3b/K2ubRNLo3dSHK +b5oIKPI1tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29m +dHdhcmUgRm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcoIJAMst +gJlaaVJbMIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6Ly90ZXN0 +Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1BggrBgEFBQcw +AYYpaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2NzcC8wQwYD +VR0fBDwwOjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0 +Y2EvcmV2b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGBACf1jFkQ9MbnKAC/ +uo17EwPxHKZfswZVpCK527LVRr33DN1DbrR5ZWchDCpV7kCOhZ+fR7sKKk22ZHSY +oH+u3PEu20J3GOB1iyY1aMNB7WvId3JvappdVWkC/VpUyFfLsGUDFuIPADmZZqCb +iJMX4loteTVfl1d4xK/1mV6Gq9MRrRqiDfpSELn+v53OM9mGspwW+NZ1CIrbCuW0 +KxZ/tPkqn8PSd9fNZR70bB7rWbnwrl+kH8xKxLl6qdlrMmg74WWwhLeQxK7+9DdP +IaDenzqx5cwWBGY/C0HcQj0gPuy3lSs1V/q+f7Y6uspPWP51PgiJLIywXS75iRAr ++UFGTzwAtyfTZSQoFyMmMULqfk6T5HtoVMqfRvPvK+mFDLWEstU1NIB1K/CRI7gI +AY65ClTU+zRS/tlF8IA7tsFvgtEf8jsI9kamlidhS1gyeg4dWcVErV4aeTPB1AUv +StPYQkKNM+NjytWHl5tNuBoDNLsc0gI/WSPiI4CIY8LwomOoiw== -----END CERTIFICATE----- diff --git a/Lib/test/keycert4.pem b/Lib/test/keycert4.pem index 53355c8..d1ebb82 100644 --- a/Lib/test/keycert4.pem +++ b/Lib/test/keycert4.pem @@ -1,73 +1,164 @@ -----BEGIN PRIVATE KEY----- -MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAK5UQiMI5VkNs2Qv -L7gUaiDdFevNUXRjU4DHAe3ZzzYLZNE69h9gO9VCSS16tJ5fT5VEu0EZyGr0e3V2 -NkX0ZoU0Hc/UaY4qx7LHmn5SYZpIxhJnkf7SyHJK1zUaGlU0/LxYqIuGCtF5dqx1 -L2OQhEx1GM6RydHdgX69G64LXcY5AgMBAAECgYAhsRMfJkb9ERLMl/oG/5sLQu9L -pWDKt6+ZwdxzlZbggQ85CMYshjLKIod2DLL/sLf2x1PRXyRG131M1E3k8zkkz6de -R1uDrIN/x91iuYzfLQZGh8bMY7Yjd2eoroa6R/7DjpElGejLxOAaDWO0ST2IFQy9 -myTGS2jSM97wcXfsSQJBANP3jelJoS5X6BRjTSneY21wcocxVuQh8pXpErALVNsT -drrFTeaBuZp7KvbtnIM5g2WRNvaxLZlAY/hXPJvi6ncCQQDSix1cebml6EmPlEZS -Mm8gwI2F9ufUunwJmBJcz826Do0ZNGByWDAM/JQZH4FX4GfAFNuj8PUb+GQfadkx -i1DPAkEA0lVsNHojvuDsIo8HGuzarNZQT2beWjJ1jdxh9t7HrTx7LIps6rb/fhOK -Zs0R6gVAJaEbcWAPZ2tFyECInAdnsQJAUjaeXXjuxFkjOFym5PvqpvhpivEx78Bu -JPTr3rAKXmfGMxxfuOa0xK1wSyshP6ZR/RBn/+lcXPKubhHQDOegwwJAJF1DBQnN -+/tLmOPULtDwfP4Zixn+/8GmGOahFoRcu6VIGHmRilJTn6MOButw7Glv2YdeC6l/ -e83Gq6ffLVfKNQ== +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQDGjpiHzq7ghxhM +ZzrnRsGBC/cmw8EREIdbqlrz/l8BFaWeipvO5Hb/MyU8xs2zLUrqIr2JNf+Eii8Y +m4bYmZclFra4jomaiSlxTZOe3dMV8m4vAq4eT2mSfZZC1+XAutqdz7WhHxhMVEm3 +AyTWvTC3qCbnlbX5VIoQUwFrsSWqDiHyaGdK3rrOTKFUKM8YPiq/BZkm6A4eiFci +5wd/SPD+w0pIscZbQW1MUr5bs54uylWaUmtfI8KJt6BDZQ/uA06c6i863sSCEI6L +gq+wyikeJGNMxZMfgu3dzfv4BiZBQX0ZhiRvqseDSdPcuVa2Ifb6CFlg298neweY +4EAIE1O+uqo5h8FF1aUOMZpQEZuzsp9R/TAMBHX1YmVjG/kRdBeaHe3whzB1Pfue +PIX2ZTMmLNYbYbfnmxhk1nn8aAvoT98pNw8y3/2k2KNsu24n9uSkkxAoqJ19WKwm +mL8MpJKAzLv45tRvhN+QLtnRdu+LJ9m29npQHFmYLbdqRfmidnMCAwEAAQKCAYBd +w1C8MRnb5W/QBJ+IP515NxFLOP2e9VM2MkgpGGH8vSAssf/Jv5GCCcD35lmU1zqd +PjKK7PjwueBrmmYfOshpN0Sp+oV4eHUdkCi5yL65inYFtRpMLewIxU2D2zgfvx0l +kMSQhYKP6O22gsGOtmCfGcTlb4kzaHyaINh25nyGxY26TxsX+/3zFbTJbUv+grzk +39vmx4aDXJbpYHfl36gOZmJZ2bl1tnvKovhJjZSRO/MYoPsbPmPLbO89ZCgVmXFc +GVkb5Cram6i3iyutSDjxWN7Fb8uy8pFLPGAXZgF7pQoXPSEHZe8GEWBnWSC9KaDa +uM9Ir847/Muy1ceCmxKcI2WrSjoH2AhPcmHgvbPE9Mynr6+uzReSP3q7Wh9PHm23 +oFx3DwdCfmjysnpAMBawNmJdWyxVDbZ6eyrhp17ADpsMaDTynZ+fJjgMr+MmWtbU +YSRD0wWtl/DrzsaePZsOjCpKYJyulC+rh9/Zz1aiwrGWPbgEAzDrD6Q1Zp0mUCEC +gcEA+XskmGIB9rRPy+YQmRgzQ555PsjLWsnQsNktP6KBhlQjFKJZXRZ0DxDTS7h8 +NrJrUDBmwfsgzggVbeO55VP5FGwD6DNeO/Bz++Fdevh8uKQFHDfk4sbIUPS91qw4 +s7OW7PR7C7Jf7Dnjmsn42o2lO4FsbcEn2F+PHOvoLl/OrSx73lS/RkdOEItW8d8/ +ExRohylnba/I2vCE9bNZd4DGjMW87j/THKPadDZWEqWggcrjY8x6ibSQGm2n2Tka +8B+vAoHBAMu+zl8kqFlYDG24dGfVpMaOYj5Osj0cV5f7O2pZ15sCevjiqoKGHH7G +O8EiI5pRBZ893+Fsx6YWmcKue88lfGvaoQjV0LUbfMfX/FoD39w/ZLx31yKEiFuc +KvMiRV5mO3kQiHBVX9vamzr5NeaErccXY9LnhaKOMX9blgiDQZH7fc3RhodcFWrC +9yfX6ryfidpPnRvK7Ops7hVnFKyyS4FaAarnzH1B2WcVcD4lYYxhMc8VXeU3eKOh +j1fI/F5ifQKBwQDpCjB670HqUzAexL9IYqSwSz3yedoK6m24ZIWx5XicI8fJJIXZ +QHoVAKB/IMtWxH8dnri+Bnj0O/TYe1pQb4pBm0xjAGjMEKYm6LNLhQXr67qiS0vQ +0eKYTKVv+9vTcLRQj2bI3Exh+wkys+tzK9DmrtS8CSvRICIs3+g4OWJzvRPP8NXj +LgQrzBzhPqpKhkvFxdVJTmSOrxFj+a5exLmzEZqT6qanIB+VYpQwQuqVkxGpTX5B +V5ssNLYPYRpapx0CgcByCtQixzcAA1u5knR9pkT76ris3YnA0Ptqk3I3XiBjoGjK +pL0CICUVBMpvmTdKai12a8DDwgqiOaZJJTchxH63NAHNGzkeFkuq5IdYrzB/bHBr +WbzukjZs6KXVv4oKg7ioVAu6rN7iBaO7x8BWzk8i0EHMzFCto1+rRM1e6HEsUBOj +v7LIU0+dmZGUGLRIbhhQPR3Yb6ZatSwyiKc23vmKZqHmUqbQOaqBm6te7beDRugF +XJVY9sqs9IJyhYpVHlUCgcAPoslwYKeAXagsxdQrH3D9VJDXVOHWKMBqQZDio5dB +Q80uWpuxtt6nhZkQO1JIWnYb6v+zbDbcgjByBIDuxCdBW9d+QQnanKmVyrXguK91 +C3OcHHOmSduFdWC3/zYW1mW97Tz1sXyam2hly1u3L5kW+GnE1hr9VVPjQNrO9+Ge +qW0coaJqKY78q3Rm2dtyZeJSFFI1o/DQ3blyItsFpg/QrR+a5XrS6Nw2ZLIL4Azo +J1CTgMwjhwlMNCI4t4dkHd0= -----END PRIVATE KEY----- Certificate: Data: - Version: 1 (0x0) - Serial Number: 12723342612721443282 (0xb09264b1f2da21d2) - Signature Algorithm: sha1WithRSAEncryption + Version: 3 (0x2) + Serial Number: + cb:2d:80:99:5a:69:52:5d + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 4 19:47:07 2013 GMT - Not After : Nov 13 19:47:07 2022 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Jul 7 14:23:16 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=fakehostname Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (1024 bit) + Public-Key: (3072 bit) Modulus: - 00:ae:54:42:23:08:e5:59:0d:b3:64:2f:2f:b8:14: - 6a:20:dd:15:eb:cd:51:74:63:53:80:c7:01:ed:d9: - cf:36:0b:64:d1:3a:f6:1f:60:3b:d5:42:49:2d:7a: - b4:9e:5f:4f:95:44:bb:41:19:c8:6a:f4:7b:75:76: - 36:45:f4:66:85:34:1d:cf:d4:69:8e:2a:c7:b2:c7: - 9a:7e:52:61:9a:48:c6:12:67:91:fe:d2:c8:72:4a: - d7:35:1a:1a:55:34:fc:bc:58:a8:8b:86:0a:d1:79: - 76:ac:75:2f:63:90:84:4c:75:18:ce:91:c9:d1:dd: - 81:7e:bd:1b:ae:0b:5d:c6:39 + 00:c6:8e:98:87:ce:ae:e0:87:18:4c:67:3a:e7:46: + c1:81:0b:f7:26:c3:c1:11:10:87:5b:aa:5a:f3:fe: + 5f:01:15:a5:9e:8a:9b:ce:e4:76:ff:33:25:3c:c6: + cd:b3:2d:4a:ea:22:bd:89:35:ff:84:8a:2f:18:9b: + 86:d8:99:97:25:16:b6:b8:8e:89:9a:89:29:71:4d: + 93:9e:dd:d3:15:f2:6e:2f:02:ae:1e:4f:69:92:7d: + 96:42:d7:e5:c0:ba:da:9d:cf:b5:a1:1f:18:4c:54: + 49:b7:03:24:d6:bd:30:b7:a8:26:e7:95:b5:f9:54: + 8a:10:53:01:6b:b1:25:aa:0e:21:f2:68:67:4a:de: + ba:ce:4c:a1:54:28:cf:18:3e:2a:bf:05:99:26:e8: + 0e:1e:88:57:22:e7:07:7f:48:f0:fe:c3:4a:48:b1: + c6:5b:41:6d:4c:52:be:5b:b3:9e:2e:ca:55:9a:52: + 6b:5f:23:c2:89:b7:a0:43:65:0f:ee:03:4e:9c:ea: + 2f:3a:de:c4:82:10:8e:8b:82:af:b0:ca:29:1e:24: + 63:4c:c5:93:1f:82:ed:dd:cd:fb:f8:06:26:41:41: + 7d:19:86:24:6f:aa:c7:83:49:d3:dc:b9:56:b6:21: + f6:fa:08:59:60:db:df:27:7b:07:98:e0:40:08:13: + 53:be:ba:aa:39:87:c1:45:d5:a5:0e:31:9a:50:11: + 9b:b3:b2:9f:51:fd:30:0c:04:75:f5:62:65:63:1b: + f9:11:74:17:9a:1d:ed:f0:87:30:75:3d:fb:9e:3c: + 85:f6:65:33:26:2c:d6:1b:61:b7:e7:9b:18:64:d6: + 79:fc:68:0b:e8:4f:df:29:37:0f:32:df:fd:a4:d8: + a3:6c:bb:6e:27:f6:e4:a4:93:10:28:a8:9d:7d:58: + ac:26:98:bf:0c:a4:92:80:cc:bb:f8:e6:d4:6f:84: + df:90:2e:d9:d1:76:ef:8b:27:d9:b6:f6:7a:50:1c: + 59:98:2d:b7:6a:45:f9:a2:76:73 Exponent: 65537 (0x10001) - Signature Algorithm: sha1WithRSAEncryption - ad:45:8a:8e:ef:c6:ef:04:41:5c:2c:4a:84:dc:02:76:0c:d0: - 66:0f:f0:16:04:58:4d:fd:68:b7:b8:d3:a8:41:a5:5c:3c:6f: - 65:3c:d1:f8:ce:43:35:e7:41:5f:53:3d:c9:2c:c3:7d:fc:56: - 4a:fa:47:77:38:9d:bb:97:28:0a:3b:91:19:7f:bc:74:ae:15: - 6b:bd:20:36:67:45:a5:1e:79:d7:75:e6:89:5c:6d:54:84:d1: - 95:d7:a7:b4:33:3c:af:37:c4:79:8f:5e:75:dc:75:c2:18:fb: - 61:6f:2d:dc:38:65:5b:ba:67:28:d0:88:d7:8d:b9:23:5a:8e: - e8:c6:bb:db:ce:d5:b8:41:2a:ce:93:08:b6:95:ad:34:20:18: - d5:3b:37:52:74:50:0b:07:2c:b0:6d:a4:4c:7b:f4:e0:fd:d1: - af:17:aa:20:cd:62:e3:f0:9d:37:69:db:41:bd:d4:1c:fb:53: - 20:da:88:9d:76:26:67:ce:01:90:a7:80:1d:a9:5b:39:73:68: - 54:0a:d1:2a:03:1b:8f:3c:43:5d:5d:c4:51:f1:a7:e7:11:da: - 31:2c:49:06:af:04:f4:b8:3c:99:c4:20:b9:06:36:a2:00:92: - 61:1d:0c:6d:24:05:e2:82:e1:47:db:a0:5f:ba:b9:fb:ba:fa: - 49:12:1e:ce + X509v3 extensions: + X509v3 Subject Alternative Name: + DNS:fakehostname + X509v3 Key Usage: critical + Digital Signature, Key Encipherment + X509v3 Extended Key Usage: + TLS Web Server Authentication, TLS Web Client Authentication + X509v3 Basic Constraints: critical + CA:FALSE + X509v3 Subject Key Identifier: + 52:E0:93:AA:52:55:B7:BB:E7:A8:E0:8C:DE:41:2E:F4:07:F0:36:FB + X509v3 Authority Key Identifier: + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 + DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server + serial:CB:2D:80:99:5A:69:52:5B + + Authority Information Access: + CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer + OCSP - URI:http://testca.pythontest.net/testca/ocsp/ + + X509v3 CRL Distribution Points: + + Full Name: + URI:http://testca.pythontest.net/testca/revocation.crl + + Signature Algorithm: sha256WithRSAEncryption + 29:d2:3f:82:3f:c1:38:35:a6:bd:81:10:fe:64:ec:ff:7e:e1: + c6:6f:7f:86:65:f9:31:6f:fb:ef:32:4e:2f:87:c8:42:de:6c: + 8d:b8:06:08:8f:37:70:95:7d:e1:40:d4:82:2b:8d:b3:4a:fd: + 34:c5:9e:df:ff:01:53:4a:4f:08:f4:58:e1:74:fc:78:e3:3e: + 71:a7:5e:66:07:ea:d2:04:31:e2:75:a8:4c:80:17:86:92:20: + d2:32:a7:9a:65:8b:1a:5f:f1:4c:c8:50:6d:00:fc:99:bf:69: + b3:48:f3:45:5a:ee:35:50:14:b8:f3:92:92:c6:9f:0e:5d:eb: + 0d:e8:ec:f2:a4:09:6b:dc:66:2b:fc:df:4c:fc:65:a1:ae:d3: + b5:88:6a:a4:e7:08:1c:94:49:e0:b8:c1:04:8c:21:09:6c:55: + 4b:2c:97:10:f1:8c:6c:d0:bb:ba:8d:93:e8:47:8b:4d:8e:7d: + 7d:85:53:18:c8:f8:f4:8f:67:3a:b1:aa:3e:18:34:6c:3a:e6: + a6:c7:2f:be:83:38:f5:d5:e5:d2:17:28:61:6c:b6:49:99:21: + 69:a4:a8:b6:94:76:fd:18:ad:35:52:45:64:fb:f1:5d:8e:bb: + c0:21:2e:59:55:24:af:bb:8f:b2:0a:7b:17:f0:34:97:8e:68: + a9:f2:d0:3e:f6:73:82:f8:7c:4e:9a:70:7d:d6:b3:8c:cc:85: + 04:5c:02:8f:74:da:88:3a:20:a8:7e:c2:9e:b0:dd:56:1f:ce: + cd:42:16:c6:14:91:ad:30:e0:dc:76:f2:2c:56:ea:38:45:d8: + c0:3e:b8:90:fa:f3:38:99:ec:44:07:35:8f:69:62:0c:f9:ef: + b7:9d:e5:15:42:6e:fb:fe:4c:ff:e8:94:5a:1a:b0:80:b2:0e: + 17:3d:e1:87:a8:08:84:93:74:68:8d:29:df:ca:0b:6a:44:32: + 8a:51:3b:d6:38:db:bd:e3:2a:1b:5e:20:48:81:82:19:91:c6: + 87:8c:0f:cd:51:ea -----BEGIN CERTIFICATE----- -MIICpzCCAY8CCQCwkmSx8toh0jANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJY -WTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNV -BAMMDW91ci1jYS1zZXJ2ZXIwHhcNMTMwMTA0MTk0NzA3WhcNMjIxMTEzMTk0NzA3 -WjBiMQswCQYDVQQGEwJYWTEXMBUGA1UEBxMOQ2FzdGxlIEFudGhyYXgxIzAhBgNV -BAoTGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRUwEwYDVQQDEwxmYWtlaG9z -dG5hbWUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAK5UQiMI5VkNs2QvL7gU -aiDdFevNUXRjU4DHAe3ZzzYLZNE69h9gO9VCSS16tJ5fT5VEu0EZyGr0e3V2NkX0 -ZoU0Hc/UaY4qx7LHmn5SYZpIxhJnkf7SyHJK1zUaGlU0/LxYqIuGCtF5dqx1L2OQ -hEx1GM6RydHdgX69G64LXcY5AgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAK1Fio7v -xu8EQVwsSoTcAnYM0GYP8BYEWE39aLe406hBpVw8b2U80fjOQzXnQV9TPcksw338 -Vkr6R3c4nbuXKAo7kRl/vHSuFWu9IDZnRaUeedd15olcbVSE0ZXXp7QzPK83xHmP -XnXcdcIY+2FvLdw4ZVu6ZyjQiNeNuSNajujGu9vO1bhBKs6TCLaVrTQgGNU7N1J0 -UAsHLLBtpEx79OD90a8XqiDNYuPwnTdp20G91Bz7UyDaiJ12JmfOAZCngB2pWzlz -aFQK0SoDG488Q11dxFHxp+cR2jEsSQavBPS4PJnEILkGNqIAkmEdDG0kBeKC4Ufb -oF+6ufu6+kkSHs4= +MIIF9zCCBF+gAwIBAgIJAMstgJlaaVJdMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV +BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA3MDcx +NDIzMTZaMGIxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZh +a2Vob3N0bmFtZTCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAMaOmIfO +ruCHGExnOudGwYEL9ybDwREQh1uqWvP+XwEVpZ6Km87kdv8zJTzGzbMtSuoivYk1 +/4SKLxibhtiZlyUWtriOiZqJKXFNk57d0xXybi8Crh5PaZJ9lkLX5cC62p3PtaEf +GExUSbcDJNa9MLeoJueVtflUihBTAWuxJaoOIfJoZ0reus5MoVQozxg+Kr8FmSbo +Dh6IVyLnB39I8P7DSkixxltBbUxSvluzni7KVZpSa18jwom3oENlD+4DTpzqLzre +xIIQjouCr7DKKR4kY0zFkx+C7d3N+/gGJkFBfRmGJG+qx4NJ09y5VrYh9voIWWDb +3yd7B5jgQAgTU766qjmHwUXVpQ4xmlARm7Oyn1H9MAwEdfViZWMb+RF0F5od7fCH +MHU9+548hfZlMyYs1htht+ebGGTWefxoC+hP3yk3DzLf/aTYo2y7bif25KSTECio +nX1YrCaYvwykkoDMu/jm1G+E35Au2dF274sn2bb2elAcWZgtt2pF+aJ2cwIDAQAB +o4IBwzCCAb8wFwYDVR0RBBAwDoIMZmFrZWhvc3RuYW1lMA4GA1UdDwEB/wQEAwIF +oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd +BgNVHQ4EFgQUUuCTqlJVt7vnqOCM3kEu9AfwNvswfQYDVR0jBHYwdIAU3b/K2ubR +NLo3dSHKb5oIKPI1tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRo +b24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZl +coIJAMstgJlaaVJbMIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6 +Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1Bggr +BgEFBQcwAYYpaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2Nz +cC8wQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5l +dC90ZXN0Y2EvcmV2b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGBACnSP4I/ +wTg1pr2BEP5k7P9+4cZvf4Zl+TFv++8yTi+HyELebI24BgiPN3CVfeFA1IIrjbNK +/TTFnt//AVNKTwj0WOF0/HjjPnGnXmYH6tIEMeJ1qEyAF4aSINIyp5plixpf8UzI +UG0A/Jm/abNI80Va7jVQFLjzkpLGnw5d6w3o7PKkCWvcZiv830z8ZaGu07WIaqTn +CByUSeC4wQSMIQlsVUsslxDxjGzQu7qNk+hHi02OfX2FUxjI+PSPZzqxqj4YNGw6 +5qbHL76DOPXV5dIXKGFstkmZIWmkqLaUdv0YrTVSRWT78V2Ou8AhLllVJK+7j7IK +exfwNJeOaKny0D72c4L4fE6acH3Ws4zMhQRcAo902og6IKh+wp6w3VYfzs1CFsYU +ka0w4Nx28ixW6jhF2MA+uJD68ziZ7EQHNY9pYgz577ed5RVCbvv+TP/olFoasICy +Dhc94YeoCISTdGiNKd/KC2pEMopRO9Y4273jKhteIEiBghmRxoeMD81R6g== -----END CERTIFICATE----- diff --git a/Lib/test/make_ssl_certs.py b/Lib/test/make_ssl_certs.py index a1f298d..ca2c12c 100644 --- a/Lib/test/make_ssl_certs.py +++ b/Lib/test/make_ssl_certs.py @@ -50,7 +50,7 @@ req_template = """ dir = cadir database = $dir/index.txt crlnumber = $dir/crl.txt - default_md = sha1 + default_md = sha256 default_days = 3600 default_crl_days = 3600 certificate = pycacert.pem @@ -88,7 +88,9 @@ req_template = """ here = os.path.abspath(os.path.dirname(__file__)) -def make_cert_key(hostname, sign=False, extra_san=''): + +def make_cert_key(hostname, sign=False, extra_san='', + ext='req_x509_extensions_full', key='rsa:3072'): print("creating cert for " + hostname) tempnames = [] for i in range(3): @@ -146,7 +148,7 @@ def make_ca(): t.flush() with tempfile.NamedTemporaryFile() as f: args = ['req', '-new', '-days', '3650', '-extensions', 'v3_ca', '-nodes', - '-newkey', 'rsa:2048', '-keyout', 'pycakey.pem', + '-newkey', 'rsa:3072', '-keyout', 'pycakey.pem', '-out', f.name, '-subj', '/C=XY/L=Castle Anthrax/O=Python Software Foundation CA/CN=our-ca-server'] check_call(['openssl'] + args) diff --git a/Lib/test/pycacert.pem b/Lib/test/pycacert.pem index 09b1f3e..73150c9 100644 --- a/Lib/test/pycacert.pem +++ b/Lib/test/pycacert.pem @@ -1,78 +1,99 @@ Certificate: Data: Version: 3 (0x2) - Serial Number: 12723342612721443280 (0xb09264b1f2da21d0) - Signature Algorithm: sha1WithRSAEncryption + Serial Number: + cb:2d:80:99:5a:69:52:5b + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 4 19:47:07 2013 GMT - Not After : Jan 2 19:47:07 2023 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Aug 26 14:23:16 2028 GMT Subject: C=XY, O=Python Software Foundation CA, CN=our-ca-server Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (2048 bit) + Public-Key: (3072 bit) Modulus: - 00:e7:de:e9:e3:0c:9f:00:b6:a1:fd:2b:5b:96:d2: - 6f:cc:e0:be:86:b9:20:5e:ec:03:7a:55:ab:ea:a4: - e9:f9:49:85:d2:66:d5:ed:c7:7a:ea:56:8e:2d:8f: - e7:42:e2:62:28:a9:9f:d6:1b:8e:eb:b5:b4:9c:9f: - 14:ab:df:e6:94:8b:76:1d:3e:6d:24:61:ed:0c:bf: - 00:8a:61:0c:df:5c:c8:36:73:16:00:cd:47:ba:6d: - a4:a4:74:88:83:23:0a:19:fc:09:a7:3c:4a:4b:d3: - e7:1d:2d:e4:ea:4c:54:21:f3:26:db:89:37:18:d4: - 02:bb:40:32:5f:a4:ff:2d:1c:f7:d4:bb:ec:8e:cf: - 5c:82:ac:e6:7c:08:6c:48:85:61:07:7f:25:e0:5c: - e0:bc:34:5f:e0:b9:04:47:75:c8:47:0b:8d:bc:d6: - c8:68:5f:33:83:62:d2:20:44:35:b1:ad:81:1a:8a: - cd:bc:35:b0:5c:8b:47:d6:18:e9:9c:18:97:cc:01: - 3c:29:cc:e8:1e:e4:e4:c1:b8:de:e7:c2:11:18:87: - 5a:93:34:d8:a6:25:f7:14:71:eb:e4:21:a2:d2:0f: - 2e:2e:d4:62:00:35:d3:d6:ef:5c:60:4b:4c:a9:14: - e2:dd:15:58:46:37:33:26:b7:e7:2e:5d:ed:42:e4: - c5:4d + 00:97:ed:55:41:ba:36:17:95:db:71:1c:d3:e1:61: + ac:58:73:e3:c6:96:cf:2b:1f:b8:08:f5:9d:4b:4b: + c7:30:f6:b8:0b:b3:52:72:a0:bb:c9:4d:3b:8e:df: + 22:8e:01:57:81:c9:92:73:cc:00:c6:ec:70:b0:3a: + 17:40:c1:df:f2:8c:36:4c:c4:a7:81:e7:b6:24:68: + e2:a0:7e:35:07:2f:a0:5b:f9:45:46:f7:1e:f0:46: + 11:fe:ca:1a:3c:50:f1:26:a9:5f:9c:22:9c:f8:41: + e1:df:4f:12:95:19:2f:5c:90:01:17:6e:7e:3e:7d: + cf:e9:09:af:25:f8:f8:42:77:2d:6d:5f:36:f2:78: + 1e:7d:4a:87:68:63:6c:06:71:1b:8d:fa:25:fe:d4: + d3:f5:a5:17:b1:ef:ea:17:cb:54:c8:27:99:80:cb: + 3c:45:f1:2c:52:1c:dd:1f:51:45:20:50:1e:5e:ab: + 57:73:1b:41:78:96:de:84:a4:7a:dd:8f:30:85:36: + 58:79:76:a0:d2:61:c8:1b:a9:94:99:63:c6:ee:f8: + 14:bf:b4:52:56:31:97:fa:eb:ac:53:9e:95:ce:4c: + c4:5a:4a:b7:ca:03:27:5b:35:57:ce:02:dc:ec:ca: + 69:f8:8a:5a:39:cb:16:20:15:03:24:61:6c:f4:7a: + fc:b6:48:e5:59:10:5c:49:d0:23:9f:fb:71:5e:3a: + e9:68:9f:34:72:80:27:b6:3f:4c:b1:d9:db:63:7f: + 67:68:4a:6e:11:f8:e8:c0:f4:5a:16:39:53:0b:68: + de:77:fa:45:e7:f8:91:cd:78:cd:28:94:97:71:54: + fb:cf:f0:37:de:c9:26:c5:dc:1b:9e:89:6d:09:ac: + c8:44:71:cb:6d:f1:97:31:d5:4c:20:33:bf:75:4a: + a0:e0:dc:69:11:ed:2a:b4:64:10:11:30:8b:0e:b0: + a7:10:d8:8a:c5:aa:1b:c8:26:8a:25:e7:66:9f:a5: + 6a:1a:2f:7c:5f:83:c6:78:4f:1f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: - BC:DD:62:D9:76:DA:1B:D2:54:6B:CF:E0:66:9B:1E:1E:7B:56:0C:0B + DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 X509v3 Authority Key Identifier: - keyid:BC:DD:62:D9:76:DA:1B:D2:54:6B:CF:E0:66:9B:1E:1E:7B:56:0C:0B + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 X509v3 Basic Constraints: CA:TRUE - Signature Algorithm: sha1WithRSAEncryption - 7d:0a:f5:cb:8d:d3:5d:bd:99:8e:f8:2b:0f:ba:eb:c2:d9:a6: - 27:4f:2e:7b:2f:0e:64:d8:1c:35:50:4e:ee:fc:90:b9:8d:6d: - a8:c5:c6:06:b0:af:f3:2d:bf:3b:b8:42:07:dd:18:7d:6d:95: - 54:57:85:18:60:47:2f:eb:78:1b:f9:e8:17:fd:5a:0d:87:17: - 28:ac:4c:6a:e6:bc:29:f4:f4:55:70:29:42:de:85:ea:ab:6c: - 23:06:64:30:75:02:8e:53:bc:5e:01:33:37:cc:1e:cd:b8:a4: - fd:ca:e4:5f:65:3b:83:1c:86:f1:55:02:a0:3a:8f:db:91:b7: - 40:14:b4:e7:8d:d2:ee:73:ba:e3:e5:34:2d:bc:94:6f:4e:24: - 06:f7:5f:8b:0e:a7:8e:6b:de:5e:75:f4:32:9a:50:b1:44:33: - 9a:d0:05:e2:78:82:ff:db:da:8a:63:eb:a9:dd:d1:bf:a0:61: - ad:e3:9e:8a:24:5d:62:0e:e7:4c:91:7f:ef:df:34:36:3b:2f: - 5d:f5:84:b2:2f:c4:6d:93:96:1a:6f:30:28:f1:da:12:9a:64: - b4:40:33:1d:bd:de:2b:53:a8:ea:be:d6:bc:4e:96:f5:44:fb: - 32:18:ae:d5:1f:f6:69:af:b6:4e:7b:1d:58:ec:3b:a9:53:a3: - 5e:58:c8:9e + Signature Algorithm: sha256WithRSAEncryption + 33:6a:54:d3:6b:c0:d7:01:5f:9d:f4:05:c1:93:66:90:50:d0: + b7:18:e9:b0:1e:4a:a0:b6:da:76:93:af:84:db:ad:15:54:31: + 15:13:e4:de:7e:4e:0c:d5:09:1c:34:35:b6:e5:4c:d6:6f:65: + 7d:32:5f:eb:fc:a9:6b:07:f7:49:82:e5:81:7e:07:80:9a:63: + f8:2c:c3:40:bc:8f:d4:2a:da:3e:d1:ee:08:b7:4d:a7:84:ca: + f4:3f:a1:98:45:be:b1:05:69:e7:df:d7:99:ab:1b:ee:8b:30: + cc:f7:fc:e7:d4:0b:17:ae:97:bf:e4:7b:fd:0f:a7:b4:85:79: + e3:59:e2:16:87:bf:1f:29:45:2c:23:93:76:be:c0:87:1d:de: + ec:2b:42:6a:e5:bb:c8:f4:0a:4a:08:0a:8c:5c:d8:7d:4d:d1: + b8:bf:d5:f7:29:ed:92:d1:94:04:e8:35:06:57:7f:2c:23:97: + 87:a5:35:8d:26:d3:1a:47:f2:16:d7:d9:c6:d4:1f:23:43:d3: + 26:99:39:ca:20:f4:71:23:6f:0c:4a:76:76:f7:76:1f:b3:fe: + bf:47:b0:fc:2a:56:81:e1:d2:dd:ee:08:d8:f4:ff:5a:dc:25: + 61:8a:91:02:b9:86:1c:f2:50:73:76:25:35:fc:b6:25:26:15: + cb:eb:c4:2b:61:0c:1c:e7:ee:2f:17:9b:ec:f0:d4:a1:84:e7: + d2:af:de:e4:1b:24:14:a7:01:87:e3:ab:29:58:46:a0:d9:c0: + 0a:e0:8d:d7:59:d3:1b:f8:54:20:3e:78:a5:a5:c8:4f:8b:03: + c4:96:9f:ec:fb:47:cf:76:2d:8d:65:34:27:bf:fa:ae:01:05: + 8a:f3:92:0a:dd:89:6c:97:a1:c7:e7:60:51:e7:ac:eb:4b:7d: + 2c:b8:65:c9:fe:5d:6a:48:55:8e:e4:c7:f9:6a:40:e1:b8:64: + 45:e9:b5:59:29:a5:5f:cf:7d:58:7d:64:79:e5:a4:09:ac:1e: + 76:65:3d:94:c4:68 -----BEGIN CERTIFICATE----- -MIIDbTCCAlWgAwIBAgIJALCSZLHy2iHQMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIEbTCCAtWgAwIBAgIJAMstgJlaaVJbMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xMzAxMDQxOTQ3MDdaFw0yMzAxMDIx -OTQ3MDdaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg -Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAOfe6eMMnwC2of0rW5bSb8zgvoa5IF7sA3pV -q+qk6flJhdJm1e3HeupWji2P50LiYiipn9Ybjuu1tJyfFKvf5pSLdh0+bSRh7Qy/ -AIphDN9cyDZzFgDNR7ptpKR0iIMjChn8Cac8SkvT5x0t5OpMVCHzJtuJNxjUArtA -Ml+k/y0c99S77I7PXIKs5nwIbEiFYQd/JeBc4Lw0X+C5BEd1yEcLjbzWyGhfM4Ni -0iBENbGtgRqKzbw1sFyLR9YY6ZwYl8wBPCnM6B7k5MG43ufCERiHWpM02KYl9xRx -6+QhotIPLi7UYgA109bvXGBLTKkU4t0VWEY3Mya35y5d7ULkxU0CAwEAAaNQME4w -HQYDVR0OBBYEFLzdYtl22hvSVGvP4GabHh57VgwLMB8GA1UdIwQYMBaAFLzdYtl2 -2hvSVGvP4GabHh57VgwLMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB -AH0K9cuN0129mY74Kw+668LZpidPLnsvDmTYHDVQTu78kLmNbajFxgawr/Mtvzu4 -QgfdGH1tlVRXhRhgRy/reBv56Bf9Wg2HFyisTGrmvCn09FVwKULeheqrbCMGZDB1 -Ao5TvF4BMzfMHs24pP3K5F9lO4MchvFVAqA6j9uRt0AUtOeN0u5zuuPlNC28lG9O -JAb3X4sOp45r3l519DKaULFEM5rQBeJ4gv/b2opj66nd0b+gYa3jnookXWIO50yR -f+/fNDY7L131hLIvxG2TlhpvMCjx2hKaZLRAMx293itTqOq+1rxOlvVE+zIYrtUf -9mmvtk57HVjsO6lTo15YyJ4= +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA4MjYx +NDIzMTZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg +Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCAaIwDQYJKoZI +hvcNAQEBBQADggGPADCCAYoCggGBAJftVUG6NheV23Ec0+FhrFhz48aWzysfuAj1 +nUtLxzD2uAuzUnKgu8lNO47fIo4BV4HJknPMAMbscLA6F0DB3/KMNkzEp4HntiRo +4qB+NQcvoFv5RUb3HvBGEf7KGjxQ8SapX5winPhB4d9PEpUZL1yQARdufj59z+kJ +ryX4+EJ3LW1fNvJ4Hn1Kh2hjbAZxG436Jf7U0/WlF7Hv6hfLVMgnmYDLPEXxLFIc +3R9RRSBQHl6rV3MbQXiW3oSket2PMIU2WHl2oNJhyBuplJljxu74FL+0UlYxl/rr +rFOelc5MxFpKt8oDJ1s1V84C3OzKafiKWjnLFiAVAyRhbPR6/LZI5VkQXEnQI5/7 +cV466WifNHKAJ7Y/TLHZ22N/Z2hKbhH46MD0WhY5Uwto3nf6Ref4kc14zSiUl3FU ++8/wN97JJsXcG56JbQmsyERxy23xlzHVTCAzv3VKoODcaRHtKrRkEBEwiw6wpxDY +isWqG8gmiiXnZp+lahovfF+DxnhPHwIDAQABo1AwTjAdBgNVHQ4EFgQU3b/K2ubR +NLo3dSHKb5oIKPI1tkgwHwYDVR0jBBgwFoAU3b/K2ubRNLo3dSHKb5oIKPI1tkgw +DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAYEAM2pU02vA1wFfnfQFwZNm +kFDQtxjpsB5KoLbadpOvhNutFVQxFRPk3n5ODNUJHDQ1tuVM1m9lfTJf6/ypawf3 +SYLlgX4HgJpj+CzDQLyP1CraPtHuCLdNp4TK9D+hmEW+sQVp59/Xmasb7oswzPf8 +59QLF66Xv+R7/Q+ntIV541niFoe/HylFLCOTdr7Ahx3e7CtCauW7yPQKSggKjFzY +fU3RuL/V9yntktGUBOg1Bld/LCOXh6U1jSbTGkfyFtfZxtQfI0PTJpk5yiD0cSNv +DEp2dvd2H7P+v0ew/CpWgeHS3e4I2PT/WtwlYYqRArmGHPJQc3YlNfy2JSYVy+vE +K2EMHOfuLxeb7PDUoYTn0q/e5BskFKcBh+OrKVhGoNnACuCN11nTG/hUID54paXI +T4sDxJaf7PtHz3YtjWU0J7/6rgEFivOSCt2JbJehx+dgUees60t9LLhlyf5dakhV +juTH+WpA4bhkRem1WSmlX899WH1keeWkCawedmU9lMRo -----END CERTIFICATE----- diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index f40ff8d..ae1c07b 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -142,7 +142,7 @@ def collect_platform(info_add): info_add('platform.python_implementation', platform.python_implementation()) info_add('platform.platform', - platform.platform(aliased=True)) + platform.platform(aliased=True, terse=True)) def collect_locale(info_add): @@ -199,32 +199,74 @@ def collect_os(info_add): call_func(info_add, 'os.cpu_count', os, 'cpu_count') call_func(info_add, 'os.loadavg', os, 'getloadavg') - # Get environment variables: filter to list - # to not leak sensitive information - ENV_VARS = ( + # Environment variables used by the stdlib and tests. Don't log the full + # environment: filter to list to not leak sensitive information. + # + # HTTP_PROXY is not logged because it can contain a password. + ENV_VARS = frozenset(( + "APPDATA", + "AR", + "ARCHFLAGS", + "ARFLAGS", + "AUDIODEV", "CC", + "CFLAGS", + "COLUMNS", + "COMPUTERNAME", "COMSPEC", + "CPP", + "CPPFLAGS", "DISPLAY", + "DISTUTILS_DEBUG", "DISTUTILS_USE_SDK", "DYLD_LIBRARY_PATH", + "ENSUREPIP_OPTIONS", + "HISTORY_FILE", "HOME", "HOMEDRIVE", "HOMEPATH", + "IDLESTARTUP", "LANG", + "LDFLAGS", + "LDSHARED", "LD_LIBRARY_PATH", + "LINES", "MACOSX_DEPLOYMENT_TARGET", + "MAILCAPS", "MAKEFLAGS", + "MIXERDEV", "MSSDK", "PATH", + "PATHEXT", + "PIP_CONFIG_FILE", + "PLAT", + "POSIXLY_CORRECT", + "PY_SAX_PARSER", + "ProgramFiles", + "ProgramFiles(x86)", + "RUNNING_ON_VALGRIND", "SDK_TOOLS_BIN", + "SERVER_SOFTWARE", "SHELL", + "SOURCE_DATE_EPOCH", + "SYSTEMROOT", "TEMP", "TERM", + "TILE_LIBRARY", + "TIX_LIBRARY", "TMP", "TMPDIR", + "TRAVIS", + "TZ", "USERPROFILE", + "VIRTUAL_ENV", "WAYLAND_DISPLAY", - ) + "WINDIR", + "_PYTHON_HOST_PLATFORM", + "_PYTHON_PROJECT_BASE", + "_PYTHON_SYSCONFIGDATA_NAME", + "__PYVENV_LAUNCHER__", + )) for name, value in os.environ.items(): uname = name.upper() if (uname in ENV_VARS @@ -275,6 +317,14 @@ def collect_readline(info_add): copy_attributes(info_add, readline, 'readline.%s', attributes, formatter=format_attr) + if not hasattr(readline, "_READLINE_LIBRARY_VERSION"): + # _READLINE_LIBRARY_VERSION has been added to CPython 3.7 + doc = getattr(readline, '__doc__', '') + if 'libedit readline' in doc: + info_add('readline.library', 'libedit readline') + elif 'GNU readline' in doc: + info_add('readline.library', 'GNU readline') + def collect_gdb(info_add): import subprocess @@ -489,6 +539,34 @@ def collect_test_support(info_add): call_func(info_add, 'test_support.python_is_optimized', support, 'python_is_optimized') +def collect_cc(info_add): + import subprocess + import sysconfig + + CC = sysconfig.get_config_var('CC') + if not CC: + return + + try: + import shlex + args = shlex.split(CC) + except ImportError: + args = CC.split() + args.append('--version') + proc = subprocess.Popen(args, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + stdout = proc.communicate()[0] + if proc.returncode: + # CC --version failed: ignore error + return + + text = stdout.splitlines()[0] + text = normalize_text(text) + info_add('CC.version', text) + + def collect_info(info): error = False info_add = info.add @@ -515,6 +593,7 @@ def collect_info(info): collect_decimal, collect_testcapi, collect_resource, + collect_cc, # Collecting from tests should be last as they have side effects. collect_test_socket, diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py index 6df8f86..70c5122 100755 --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -174,20 +174,21 @@ Pattern examples: import StringIO import datetime import getopt +import imp import json +import math import os +import platform import random import re import shutil import sys +import sysconfig +import tempfile import time import traceback -import warnings import unittest -import tempfile -import imp -import platform -import sysconfig +import warnings # Some times __path__ and __file__ are not absolute (e.g. while running from @@ -240,6 +241,7 @@ SKIPPED = -2 RESOURCE_DENIED = -3 INTERRUPTED = -4 CHILD_ERROR = -5 # error in a child process +TEST_DID_NOT_RUN = -6 # error in a child process # Minimum duration of a test to display its duration or to mention that # the test is running in background @@ -248,7 +250,7 @@ PROGRESS_MIN_TIME = 30.0 # seconds # Display the running tests if nothing happened last N seconds PROGRESS_UPDATE = 30.0 # seconds -from test import test_support +from test import support ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network', 'bsddb', 'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', @@ -270,13 +272,25 @@ def usage(code, msg=''): def format_duration(seconds): - if seconds < 1.0: - return '%.0f ms' % (seconds * 1e3) - if seconds < 60.0: - return '%.0f sec' % seconds - - minutes, seconds = divmod(seconds, 60.0) - return '%.0f min %.0f sec' % (minutes, seconds) + ms = int(math.ceil(seconds * 1e3)) + seconds, ms = divmod(ms, 1000) + minutes, seconds = divmod(seconds, 60) + hours, minutes = divmod(minutes, 60) + + parts = [] + if hours: + parts.append('%s hour' % hours) + if minutes: + parts.append('%s min' % minutes) + if seconds: + parts.append('%s sec' % seconds) + if ms: + parts.append('%s ms' % ms) + if not parts: + return '0 ms' + + parts = parts[:2] + return ' '.join(parts) _FORMAT_TEST_RESULT = { @@ -287,6 +301,7 @@ _FORMAT_TEST_RESULT = { RESOURCE_DENIED: '%s skipped (resource denied)', INTERRUPTED: '%s interrupted', CHILD_ERROR: '%s crashed', + TEST_DID_NOT_RUN: '%s run no tests', } @@ -317,7 +332,7 @@ def unload_test_modules(save_modules): # Unload the newly imported modules (best effort finalization) for module in sys.modules.keys(): if module not in save_modules and module.startswith("test."): - test_support.unload(module) + support.unload(module) def main(tests=None, testdir=None, verbose=0, quiet=False, @@ -350,7 +365,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, """ regrtest_start_time = time.time() - test_support.record_original_stdout(sys.stdout) + support.record_original_stdout(sys.stdout) try: opts, args = getopt.getopt(sys.argv[1:], 'hvqxsSrf:lu:t:TD:NLR:FwWM:j:PGm:', ['help', 'verbose', 'verbose2', 'verbose3', 'quiet', @@ -406,7 +421,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, elif o == '--matchfile': if match_tests is None: match_tests = [] - filename = os.path.join(test_support.SAVEDCWD, a) + filename = os.path.join(support.SAVEDCWD, a) with open(filename) as fp: for line in fp: match_tests.append(line.strip()) @@ -439,7 +454,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, if len(huntrleaks) == 2 or not huntrleaks[2]: huntrleaks[2:] = ["reflog.txt"] elif o in ('-M', '--memlimit'): - test_support.set_memlimit(a) + support.set_memlimit(a) elif o in ('-u', '--use'): u = [x.lower() for x in a.split(',')] for r in u: @@ -503,6 +518,15 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, except ValueError: pass + if huntrleaks: + warmup, repetitions, _ = huntrleaks + if warmup < 1 or repetitions < 1: + msg = ("Invalid values for the --huntrleaks/-R parameters. The " + "number of warmups and repetitions must be at least 1 " + "each (1:1).") + print >>sys.stderr, msg + sys.exit(2) + if slaveargs is not None: args, kwargs = json.loads(slaveargs) if kwargs['huntrleaks']: @@ -525,6 +549,9 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, skipped = [] resource_denieds = [] environment_changed = [] + rerun = [] + run_no_tests = [] + first_result = None interrupted = False if findleaks: @@ -552,7 +579,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, if fromfile: tests = [] - fp = open(os.path.join(test_support.SAVEDCWD, fromfile)) + fp = open(os.path.join(support.SAVEDCWD, fromfile)) for line in fp: guts = line.split() # assuming no test has whitespace in its name if guts and not guts[0].startswith('#'): @@ -572,6 +599,12 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, nottests.add(arg) args = [] + if huntrleaks: + # FIXME: bpo-31731: test_io hangs with --huntrleaks + print("Warning: bpo-31731: test_io hangs with --huntrleaks: " + "exclude the test") + nottests.add('test_io') + display_header = (verbose or header or not (quiet or single or tests or args)) and (not pgo) alltests = findtests(testdir, stdtests, nottests) selected = tests or args or alltests @@ -596,7 +629,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, tracer = trace.Trace(trace=False, count=True) test_times = [] - test_support.use_resources = use_resources + support.use_resources = use_resources save_modules = set(sys.modules) def accumulate_result(test, result): @@ -614,6 +647,8 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, elif ok == RESOURCE_DENIED: skipped.append(test) resource_denieds.append(test) + elif ok == TEST_DID_NOT_RUN: + run_no_tests.append(test) elif ok != INTERRUPTED: raise ValueError("invalid test result: %r" % ok) @@ -637,8 +672,9 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, def display_progress(test_index, test): # "[ 51/405/1] test_tcl" line = "{1:{0}}{2}".format(test_count_width, test_index, test_count) - if bad and not pgo: - line = '{}/{}'.format(line, len(bad)) + fails = len(bad) + len(environment_changed) + if fails and not pgo: + line = '{}/{}'.format(line, fails) line = '[{}]'.format(line) # add the system load prefix: "load avg: 1.80 " @@ -669,6 +705,12 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, if ncpu: print "== CPU count:", ncpu + if huntrleaks: + warmup, repetitions, _ = huntrleaks + if warmup < 3: + print("WARNING: Running tests with --huntrleaks/-R and less than " + "3 warmup repetitions can give false positives!") + if randomize: random.seed(random_seed) print "Using random seed", random_seed @@ -695,7 +737,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, ) yield (test, args_tuple) pending = tests_and_args() - opt_args = test_support.args_from_interpreter_flags() + opt_args = support.args_from_interpreter_flags() base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest'] # required to spawn a new process with PGO flag on/off if pgo: @@ -768,7 +810,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, continue dt = time.time() - worker.start_time if dt >= PROGRESS_MIN_TIME: - running.append('%s (%.0f sec)' % (current_test, dt)) + running.append('%s (%s)' % (current_test, format_duration(dt))) return running finished = 0 @@ -782,6 +824,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, running = get_running(workers) if running and not pgo: print('running: %s' % ', '.join(running)) + sys.stdout.flush() continue test, stdout, stderr, result = item @@ -795,14 +838,14 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, if (ok not in (CHILD_ERROR, INTERRUPTED) and test_time >= PROGRESS_MIN_TIME and not pgo): - text += ' (%.0f sec)' % test_time + text += ' (%s)' % format_duration(test_time) running = get_running(workers) if running and not pgo: text += ' -- running: %s' % ', '.join(running) display_progress(test_index, text) if stdout: - print stdout + print(stdout) sys.stdout.flush() if stderr and not pgo: print >>sys.stderr, stderr @@ -880,56 +923,109 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, unload_test_modules(save_modules) - if interrupted and not pgo: + + def get_tests_result(): + result = [] + if bad: + result.append("FAILURE") + elif fail_env_changed and environment_changed: + result.append("ENV CHANGED") + elif not any((good, bad, skipped, interrupted, environment_changed)): + result.append("NO TEST RUN") + + if interrupted: + result.append("INTERRUPTED") + + if not result: + result.append("SUCCESS") + + result = ', '.join(result) + if first_result: + result = '%s then %s' % (first_result, result) + return result + + + def display_result(): # print a newline after ^C print - print "Test suite interrupted by signal SIGINT." - omitted = set(selected) - set(good) - set(bad) - set(skipped) - print count(len(omitted), "test"), "omitted:" - printlist(omitted) - if good and not quiet and not pgo: - if not bad and not skipped and not interrupted and len(good) > 1: - print "All", - print count(len(good), "test"), "OK." - if print_slow: - test_times.sort(reverse=True) - print "10 slowest tests:" - for test_time, test in test_times[:10]: - print("- %s: %.1fs" % (test, test_time)) - if bad and not pgo: - print count(len(bad), "test"), "failed:" - printlist(bad) - if environment_changed and not pgo: - print "{} altered the execution environment:".format( - count(len(environment_changed), "test")) - printlist(environment_changed) - if skipped and not quiet and not pgo: - print count(len(skipped), "test"), "skipped:" - printlist(skipped) - - e = _ExpectedSkips() - plat = sys.platform - if e.isvalid(): - surprise = set(skipped) - e.getexpected() - set(resource_denieds) - if surprise: - print count(len(surprise), "skip"), \ - "unexpected on", plat + ":" - printlist(surprise) + print("== Tests result: %s ==" % get_tests_result()) + + if interrupted and not pgo: + print + print "Test suite interrupted by signal SIGINT." + omitted = set(selected) - set(good) - set(bad) - set(skipped) + print count(len(omitted), "test"), "omitted:" + printlist(omitted) + + if good and not quiet and not pgo: + print + if not bad and not skipped and not interrupted and len(good) > 1: + print "All", + print count(len(good), "test"), "OK." + + if print_slow: + test_times.sort(reverse=True) + print + print "10 slowest tests:" + for test_time, test in test_times[:10]: + print("- %s: %.1fs" % (test, test_time)) + + if bad and not pgo: + print + print count(len(bad), "test"), "failed:" + printlist(bad) + + if environment_changed and not pgo: + print + print "{} altered the execution environment:".format( + count(len(environment_changed), "test")) + printlist(environment_changed) + + if skipped and not quiet and not pgo: + print + print count(len(skipped), "test"), "skipped:" + printlist(skipped) + + e = _ExpectedSkips() + plat = sys.platform + if e.isvalid(): + surprise = set(skipped) - e.getexpected() - set(resource_denieds) + if surprise: + print count(len(surprise), "skip"), \ + "unexpected on", plat + ":" + printlist(surprise) + else: + print "Those skips are all expected on", plat + "." else: - print "Those skips are all expected on", plat + "." - else: - print "Ask someone to teach regrtest.py about which tests are" - print "expected to get skipped on", plat + "." + print "Ask someone to teach regrtest.py about which tests are" + print "expected to get skipped on", plat + "." + + if rerun: + print("") + print("%s:" % count(len(rerun), "re-run test")) + printlist(rerun) + + if run_no_tests: + print("") + print("%s run no tests:" % count(len(run_no_tests), "test")) + printlist(run_no_tests) + + + display_result() if verbose2 and bad: + first_result = get_tests_result() + + print print "Re-running failed tests in verbose mode" - for test in bad[:]: + rerun = bad[:] + for test in rerun: print "Re-running test %r in verbose mode" % test sys.stdout.flush() try: - test_support.verbose = True + support.verbose = True ok = runtest(test, True, quiet, huntrleaks, None, pgo, - testdir=testdir) + match_tests=match_tests, testdir=testdir) except KeyboardInterrupt: # print a newline separate from the ^C print @@ -942,6 +1038,8 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, print count(len(bad), "test"), "failed again:" printlist(bad) + display_result() + if single: if next_single_test: with open(filename, 'w') as fp: @@ -960,15 +1058,7 @@ def main(tests=None, testdir=None, verbose=0, quiet=False, duration = time.time() - regrtest_start_time print("Total duration: %s" % format_duration(duration)) - if bad: - result = "FAILURE" - elif interrupted: - result = "INTERRUPTED" - elif fail_env_changed and environment_changed: - result = "ENV CHANGED" - else: - result = "SUCCESS" - print("Tests result: %s" % result) + print("Tests result: %s" % get_tests_result()) if bad: sys.exit(2) @@ -1031,15 +1121,20 @@ def runtest(test, verbose, quiet, ENV_CHANGED test failed because it changed the execution environment FAILED test failed PASSED test passed + EMPTY_TEST_SUITE test ran no subtests. """ - test_support.verbose = verbose # Tell tests to be moderately quiet + support.verbose = verbose # Tell tests to be moderately quiet if use_resources is not None: - test_support.use_resources = use_resources + support.use_resources = use_resources try: - test_support.set_match_tests(match_tests) + support.set_match_tests(match_tests) + # reset the environment_altered flag to detect if a test altered + # the environment + support.environment_altered = False if failfast: - test_support.failfast = True + support.failfast = True + return runtest_inner(test, verbose, quiet, huntrleaks, pgo, testdir) finally: cleanup_test_droppings(test, verbose) @@ -1139,31 +1234,31 @@ class saved_test_environment: asyncore.close_all(ignore_all=True) asyncore.socket_map.update(saved_map) - def get_test_support_TESTFN(self): - if os.path.isfile(test_support.TESTFN): + def get_support_TESTFN(self): + if os.path.isfile(support.TESTFN): result = 'f' - elif os.path.isdir(test_support.TESTFN): + elif os.path.isdir(support.TESTFN): result = 'd' else: result = None return result - def restore_test_support_TESTFN(self, saved_value): + def restore_support_TESTFN(self, saved_value): if saved_value is None: - if os.path.isfile(test_support.TESTFN): - os.unlink(test_support.TESTFN) - elif os.path.isdir(test_support.TESTFN): - shutil.rmtree(test_support.TESTFN) + if os.path.isfile(support.TESTFN): + os.unlink(support.TESTFN) + elif os.path.isdir(support.TESTFN): + shutil.rmtree(support.TESTFN) def get_files(self): return sorted(fn + ('/' if os.path.isdir(fn) else '') for fn in os.listdir(os.curdir)) def restore_files(self, saved_value): - fn = test_support.TESTFN + fn = support.TESTFN if fn not in saved_value and (fn + '/') not in saved_value: if os.path.isfile(fn): - test_support.unlink(fn) + support.unlink(fn) elif os.path.isdir(fn): - test_support.rmtree(fn) + support.rmtree(fn) def resource_info(self): for name in self.resources: @@ -1180,6 +1275,10 @@ class saved_test_environment: def __exit__(self, exc_type, exc_val, exc_tb): saved_values = self.saved_values del self.saved_values + + # Read support.environment_altered, set by support helper functions + self.changed |= support.environment_altered + for name, get, restore in self.resource_info(): current = get() original = saved_values.pop(name) @@ -1202,10 +1301,10 @@ class saved_test_environment: def post_test_cleanup(): - test_support.reap_children() + support.reap_children() def runtest_inner(test, verbose, quiet, huntrleaks=False, pgo=False, testdir=None): - test_support.unload(test) + support.unload(test) if verbose: capture_stdout = None else: @@ -1231,16 +1330,17 @@ def runtest_inner(test, verbose, quiet, huntrleaks=False, pgo=False, testdir=Non # being imported. For tests based on unittest or doctest, # explicitly invoke their test_main() function (if it exists). indirect_test = getattr(the_module, "test_main", None) - if indirect_test is not None: - indirect_test() if huntrleaks: refleak = dash_R(the_module, test, indirect_test, huntrleaks) + else: + if indirect_test is not None: + indirect_test() test_time = time.time() - start_time post_test_cleanup() finally: sys.stdout = save_stdout - except test_support.ResourceDenied, msg: + except support.ResourceDenied, msg: if not quiet and not pgo: print test, "skipped --", msg sys.stdout.flush() @@ -1252,11 +1352,13 @@ def runtest_inner(test, verbose, quiet, huntrleaks=False, pgo=False, testdir=Non return SKIPPED, test_time except KeyboardInterrupt: raise - except test_support.TestFailed, msg: + except support.TestFailed, msg: if not pgo: print >>sys.stderr, "test", test, "failed --", msg sys.stderr.flush() return FAILED, test_time + except support.TestDidNotRun: + return TEST_DID_NOT_RUN, test_time except: type, value = sys.exc_info()[:2] if not pgo: @@ -1297,7 +1399,7 @@ def cleanup_test_droppings(testname, verbose): # since if a test leaves a file open, it cannot be deleted by name (while # there's nothing we can do about that here either, we can display the # name of the offending test, which is a real help). - for name in (test_support.TESTFN, + for name in (support.TESTFN, "db_home", ): if not os.path.exists(name): @@ -1364,39 +1466,61 @@ def dash_R(the_module, test, indirect_test, huntrleaks): deltas = [] nwarmup, ntracked, fname = huntrleaks - fname = os.path.join(test_support.SAVEDCWD, fname) + fname = os.path.join(support.SAVEDCWD, fname) repcount = nwarmup + ntracked + rc_deltas = [0] * ntracked + fd_deltas = [0] * ntracked + print >> sys.stderr, "beginning", repcount, "repetitions" print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount] dash_R_cleanup(fs, ps, pic, zdc, abcs) + # initialize variables to make pyflakes quiet + rc_before = fd_before = 0 for i in range(repcount): - rc_before = sys.gettotalrefcount() run_the_test() sys.stderr.write('.') dash_R_cleanup(fs, ps, pic, zdc, abcs) rc_after = sys.gettotalrefcount() + fd_after = support.fd_count() if i >= nwarmup: - deltas.append(rc_after - rc_before) + rc_deltas[i - nwarmup] = rc_after - rc_before + fd_deltas[i - nwarmup] = fd_after - fd_before + rc_before = rc_after + fd_before = fd_after print >> sys.stderr - # bpo-30776: Try to ignore false positives: - # - # [3, 0, 0] - # [0, 1, 0] - # [8, -8, 1] - # - # Expected leaks: - # - # [5, 5, 6] - # [10, 1, 1] - if all(delta >= 1 for delta in deltas): - msg = '%s leaked %s references, sum=%s' % (test, deltas, sum(deltas)) - print >> sys.stderr, msg - with open(fname, "a") as refrep: - print >> refrep, msg - refrep.flush() - return True - return False + # These checkers return False on success, True on failure + def check_rc_deltas(deltas): + # Checker for reference counters and memomry blocks. + # + # bpo-30776: Try to ignore false positives: + # + # [3, 0, 0] + # [0, 1, 0] + # [8, -8, 1] + # + # Expected leaks: + # + # [5, 5, 6] + # [10, 1, 1] + return all(delta >= 1 for delta in deltas) + + def check_fd_deltas(deltas): + return any(deltas) + + failed = False + for deltas, item_name, checker in [ + (rc_deltas, 'references', check_rc_deltas), + (fd_deltas, 'file descriptors', check_fd_deltas) + ]: + if checker(deltas): + msg = '%s leaked %s %s, sum=%s' % (test, deltas, item_name, sum(deltas)) + print >> sys.stderr, msg + with open(fname, "a") as refrep: + print >> refrep, msg + refrep.flush() + failed = True + return failed def dash_R_cleanup(fs, ps, pic, zdc, abcs): import gc, copy_reg @@ -1580,12 +1704,12 @@ def _list_cases(suite): if isinstance(test, unittest.TestSuite): _list_cases(test) elif isinstance(test, unittest.TestCase): - if test_support.match_test(test): + if support.match_test(test): print(test.id()) def list_cases(testdir, selected, match_tests): - test_support.verbose = False - test_support.set_match_tests(match_tests) + support.verbose = False + support.set_match_tests(match_tests) save_modules = set(sys.modules) skipped = [] @@ -2019,8 +2143,8 @@ def main_in_temp_cwd(): # Run the tests in a context manager that temporary changes the CWD to a # temporary and writable directory. If it's not possible to create or # change the CWD, the original CWD will be used. The original CWD is - # available from test_support.SAVEDCWD. - with test_support.temp_cwd(TESTCWD, quiet=True): + # available from support.SAVEDCWD. + with support.temp_cwd(TESTCWD, quiet=True): main() if __name__ == '__main__': diff --git a/Lib/test/revocation.crl b/Lib/test/revocation.crl index 6d89b08..c05461c 100644 --- a/Lib/test/revocation.crl +++ b/Lib/test/revocation.crl @@ -1,11 +1,14 @@ -----BEGIN X509 CRL----- -MIIBpjCBjwIBATANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJYWTEmMCQGA1UE +MIICJjCBjwIBATANBgkqhkiG9w0BAQsFADBNMQswCQYDVQQGEwJYWTEmMCQGA1UE CgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNVBAMMDW91ci1j -YS1zZXJ2ZXIXDTEzMTEyMTE3MDg0N1oXDTIzMDkzMDE3MDg0N1qgDjAMMAoGA1Ud -FAQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQCNJXC2mVKauEeN3LlQ3ZtM5gkH3ExH -+i4bmJjtJn497WwvvoIeUdrmVXgJQR93RtV37hZwN0SXMLlNmUZPH4rHhihayw4m -unCzVj/OhCCY7/TPjKuJ1O/0XhaLBpBVjQN7R/1ujoRKbSia/CD3vcn7Fqxzw7LK -fSRCKRGTj1CZiuxrphtFchwALXSiFDy9mr2ZKhImcyq1PydfgEzU78APpOkMQsIC -UNJ/cf3c9emzf+dUtcMEcejQ3mynBo4eIGg1EW42bz4q4hSjzQlKcBV0muw5qXhc -HOxH2iTFhQ7SrvVuK/dM14rYM4B5mSX3nRC1kNmXpS9j3wJDhuwmjHed +YS1zZXJ2ZXIXDTE4MDgyOTE0MjMxNloXDTI4MDcwNzE0MjMxNlqgDjAMMAoGA1Ud +FAQDAgEAMA0GCSqGSIb3DQEBCwUAA4IBgQCPhrtGSbuvxPAI3YWQFDB4iOWdBnVk +ugW1lsifmCsE86FfID0EwUut1SRHlksltMtcoULMEIdu8yMLWci++4ve22EEuMKT +HUc3T/wBIuQUhA7U4deFG8CZPAxRpNoK470y7dkD4OVf0Gxa6WYDl9z8mXKmWCB9 +hvzqVfLWNSLTAVPsHtkD5PXdi5yRkQr6wYD7poWaIvkpsn7EKCY6Tw5V3rsbRuZq +AGVCq5TH3mctcmwLloCJ4Xr/1q0DsRrYxeeLYxE+UpvvCbVBKgtjBK7zINS7AbcJ +CYCYKUwGWv1fYKJ+KQQHf75mT3jQ9lWuzOj/YWK4k1EBnYmVGuKKt73lLFxC6h3y +MUnaBZc1KZSyJj0IxfHg/o6qx8NgKOl9XRIQ5g5B30cwpPOskGhEhodbTTY3bPtm +RQ36JvQZngzmkhyhr+MDEV5yUTOShfUiclzQOx26CmLmLHWxOZgXtFZob/oKrvbm +Gen/+7K7YTw6hfY52U7J2FuQRGOyzBXfBYQ= -----END X509 CRL----- diff --git a/Lib/test/ssl_cert.pem b/Lib/test/ssl_cert.pem index 47a7d7e..de59671 100644 --- a/Lib/test/ssl_cert.pem +++ b/Lib/test/ssl_cert.pem @@ -1,15 +1,26 @@ -----BEGIN CERTIFICATE----- -MIICVDCCAb2gAwIBAgIJANfHOBkZr8JOMA0GCSqGSIb3DQEBBQUAMF8xCzAJBgNV -BAYTAlhZMRcwFQYDVQQHEw5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDAeFw0xMDEw -MDgyMzAxNTZaFw0yMDEwMDUyMzAxNTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH -Ew5DYXN0bGUgQW50aHJheDEjMCEGA1UEChMaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAw -gYkCgYEA21vT5isq7F68amYuuNpSFlKDPrMUCa4YWYqZRt2OZ+/3NKaZ2xAiSwr7 -6MrQF70t5nLbSPpqE5+5VrS58SY+g/sXLiFd6AplH1wJZwh78DofbFYXUggktFMt -pTyiX8jtP66bkcPkDADA089RI1TQR6Ca+n7HFa7c1fabVV6i3zkCAwEAAaMYMBYw -FAYDVR0RBA0wC4IJbG9jYWxob3N0MA0GCSqGSIb3DQEBBQUAA4GBAHPctQBEQ4wd -BJ6+JcpIraopLn8BGhbjNWj40mmRqWB/NAWF6M5ne7KpGAu7tLeG4hb1zLaldK8G -lxy2GPSRF6LFS48dpEj2HbMv2nvv6xxalDMJ9+DicWgAKTQ6bcX2j3GUkCR0g/T1 -CRlNBAAlvhKzO7Clpf9l0YKBEfraJByX +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/ssl_key.passwd.pem b/Lib/test/ssl_key.passwd.pem index 2524672..e4f1370 100644 --- a/Lib/test/ssl_key.passwd.pem +++ b/Lib/test/ssl_key.passwd.pem @@ -1,18 +1,42 @@ -----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED -DEK-Info: DES-EDE3-CBC,1A8D9D2A02EC698A +DEK-Info: DES-EDE3-CBC,8064BE1494B24B13 -kJYbfZ8L0sfe9Oty3gw0aloNnY5E8fegRfQLZlNoxTl6jNt0nIwI8kDJ36CZgR9c -u3FDJm/KqrfUoz8vW+qEnWhSG7QPX2wWGPHd4K94Yz/FgrRzZ0DoK7XxXq9gOtVA -AVGQhnz32p+6WhfGsCr9ArXEwRZrTk/FvzEPaU5fHcoSkrNVAGX8IpSVkSDwEDQr -Gv17+cfk99UV1OCza6yKHoFkTtrC+PZU71LomBabivS2Oc4B9hYuSR2hF01wTHP+ -YlWNagZOOVtNz4oKK9x9eNQpmfQXQvPPTfusexKIbKfZrMvJoxcm1gfcZ0H/wK6P -6wmXSG35qMOOztCZNtperjs1wzEBXznyK8QmLcAJBjkfarABJX9vBEzZV0OUKhy+ -noORFwHTllphbmydLhu6ehLUZMHPhzAS5UN7srtpSN81eerDMy0RMUAwA7/PofX1 -94Me85Q8jP0PC9ETdsJcPqLzAPETEYu0ELewKRcrdyWi+tlLFrpE5KT/s5ecbl9l -7B61U4Kfd1PIXc/siINhU3A3bYK+845YyUArUOnKf1kEox7p1RpD7yFqVT04lRTo -cibNKATBusXSuBrp2G6GNuhWEOSafWCKJQAzgCYIp6ZTV2khhMUGppc/2H3CF6cO -zX0KtlPVZC7hLkB6HT8SxYUwF1zqWY7+/XPPdc37MeEZ87Q3UuZwqORLY+Z0hpgt -L5JXBCoklZhCAaN2GqwFLXtGiRSRFGY7xXIhbDTlE65Wv1WGGgDLMKGE1gOz3yAo -2jjG1+yAHJUdE69XTFHSqSkvaloA1W03LdMXZ9VuQJ/ySXCie6ABAQ== +KJrffOMbo8M0I3PzcYxRZGMpKD1yB3Ii4+bT5XoanxjIJ+4fdx6LfZ0Rsx+riyzs +tymsQu/iYY9j+4rCvN9+eetsL1X6iZpiimKsLexcid9M3fb0vxED5Sgw0dvunCUA +xhqjLIKR92MKbODHf6KrDKCpsiPbjq4gZ7P+uCGXAMHL3MXIJSC0hW9rK7Ce6oyO +CjpIcgB8x+GUWZZZhAFdlzIHMZrteNP2P5HK6QcaT71P034Dz1hhqoj4Q0t+Fta2 +4tfsM/bnTR/l6hwlhPa1e3Uj322tDTDWBScgWANn5+sEWldLmozMaWhZsn22pfk2 +KjRMGXG024JVheV882nbdOBvG7oq+lxkZ/ZP+vvqJqnvYtf7WtM8UivzYpe5Hz5b +kVvWzPjBLUSZ9whM9rDLqSSqMPyPvDTuEmLkuq+xm7pYJmsLqIMP2klZLqRxLX6K +uqwplb8UG440qauxgnQ905PId1l2fJEnRtV+7vXprA0L0QotgXLVHBhLmTFM+3PH +9H3onf31dionUAPrn3nfVE36HhvVgRyvDBnBzJSIMighgq21Qx/d1dk0DRYi1hUI +nCHl0YJPXheVcXR7JiSF2XQCAaFuS1Mr7NCXfWZOZQC/0dkvmHnl9DUAhuqq9BNZ +1cKhZXcKHadg2/r0Zup/oDzmHPUEfTAXT0xbqoWlhkdwbF2veWQ96A/ncx3ISTb4 +PkXBlX9rdia8nmtyQDQRn4NuvchbaGkj4WKFC8pF8Hn7naHqwjpHaDUimBc0CoQW +edNJqruKWwtSVLuwKHCC2gZFX9AXSKJXJz/QRSUlhFGOhuF/J6yKaXj6n5lxWNiQ +54J+OP/hz2aS95CD2+Zf1SKpxdWiLZSIQqESpmmUrXROixNJZ/Z7gI74Dd9dSJOH +W+3AU03vrrFZVrJVZhjcINHoH1Skh6JKscH18L6x4U868nSr4SrRLX8BhHllOQyD +bmU+PZAjF8ZBIaCtTGulDXD29F73MeAZeTSsgQjFu0iKLj1wPiphbx8i/SUtR4YP +X6PVA04g66r1NBw+3RQASVorZ3g1MSFvITHXcbKkBDeJH2z1+c6t/VVyTONnQhM5 +lLgRSk6HCbetvT9PKxWrWutA12pdBYEHdZhMHVf2+xclky7l09w8hg2/qqcdGRGe +oAOZ72t0l5ObNyaruDKUS6f4AjOyWq/Xj5xuFtf1n3tQHyslSyCTPcAbQhDfTHUx +vixb/V9qvYPt7OCn8py7v1M69NH42QVFAvwveDIFjZdqfIKBoJK2V4qPoevJI6uj +Q5ByMt8OXOjSXNpHXpYQWUiWeCwOEBXJX8rzCHdMtg37jJ0zCmeErR1NTdg+EujM +TWYgd06jlT67tURST0aB2kg4ijKgUJefD313LW1zC6gVsTbjSZxYyRbPfSP6flQB +yCi1C19E2OsgleqbkBVC5GlYUzaJT7SGjCRmGx1eqtbrALu+LVH24Wceexlpjydl ++s2nf/DZlKun/tlPh6YioifPCJjByZMQOCEfIox6BkemZETz8uYA4TTWimG13Z03 +gyDGC2jdpEW414J2qcQDvrdUgJ+HlhrAAHaWpMQDbXYxBGoZ+3+ORvQV4kAsCwL8 +k3EIrVpePdik+1xgOWsyLj6QxFXlTMvL6Wc5pnArFPORsgHEolJvxSPTf9aAHNPn +V2WBvxiLBtYpGrujAUM40Syx/aN2RPtcXYPAusHUBw+S8/p+/8Kg8GZmnIXG3F89 +45Eepl2quZYIrou7a1fwIpIIZ0hFiBQ1mlHVMFtxwVHS1bQb3SU2GeO+JcGjdVXc +04qeGuQ5M164eQ5C0T7ZQ1ULiUlFWKD30m+cjqmZzt3d7Q0mKpMKuESIuZJo/wpD +Nas432aLKUhcNx/pOYLkKJRpGZKOupQoD5iUj/j44o8JoFkDK33v2S57XB5QGz28 +9Zuhx49b3W8mbM6EBanlQKLWJGCxXqc/jhYhFWn+b0MhidynFgA0oeWvf6ZDyt6H +Yi5Etxsar09xp0Do3NxtQXLuSUu0ji2pQzSIKuoqQWKqldm6VrpwojiqJhy4WQBQ +aVVyFeWBC7G3Zj76dO+yp2sfJ0itJUQ8AIB9Cg0f34rEZu+r9luPmqBoUeL95Tk7 +YvCOU3Jl8Iqysv8aNpVXT8sa8rrSbruWCByEePZ37RIdHLMVBwVY0eVaFQjrjU7E +mXmM9eaoYLfXOllsQ+M2+qPFUITr/GU3Qig13DhK/+yC1R6V2a0l0WRhMltIPYKW +Ztvvr4hK5LcYCeS113BLiMbDIMMZZYGDZGMdC8DnnVbT2loF0Rfmp80Af31KmMQ4 +6XvMatW9UDjBoY5a/YMpdm7SRwm+MgV2KNPpc2kST87/yi9oprGAb8qiarHiHTM0 -----END RSA PRIVATE KEY----- diff --git a/Lib/test/ssl_key.pem b/Lib/test/ssl_key.pem index 3fd3bbd..1ea4578 100644 --- a/Lib/test/ssl_key.pem +++ b/Lib/test/ssl_key.pem @@ -1,16 +1,40 @@ -----BEGIN PRIVATE KEY----- -MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBANtb0+YrKuxevGpm -LrjaUhZSgz6zFAmuGFmKmUbdjmfv9zSmmdsQIksK++jK0Be9LeZy20j6ahOfuVa0 -ufEmPoP7Fy4hXegKZR9cCWcIe/A6H2xWF1IIJLRTLaU8ol/I7T+um5HD5AwAwNPP -USNU0Eegmvp+xxWu3NX2m1Veot85AgMBAAECgYA3ZdZ673X0oexFlq7AAmrutkHt -CL7LvwrpOiaBjhyTxTeSNWzvtQBkIU8DOI0bIazA4UreAFffwtvEuPmonDb3F+Iq -SMAu42XcGyVZEl+gHlTPU9XRX7nTOXVt+MlRRRxL6t9GkGfUAXI3XxJDXW3c0vBK -UL9xqD8cORXOfE06rQJBAP8mEX1ERkR64Ptsoe4281vjTlNfIbs7NMPkUnrn9N/Y -BLhjNIfQ3HFZG8BTMLfX7kCS9D593DW5tV4Z9BP/c6cCQQDcFzCcVArNh2JSywOQ -ZfTfRbJg/Z5Lt9Fkngv1meeGNPgIMLN8Sg679pAOOWmzdMO3V706rNPzSVMME7E5 -oPIfAkEA8pDddarP5tCvTTgUpmTFbakm0KoTZm2+FzHcnA4jRh+XNTjTOv98Y6Ik -eO5d1ZnKXseWvkZncQgxfdnMqqpj5wJAcNq/RVne1DbYlwWchT2Si65MYmmJ8t+F -0mcsULqjOnEMwf5e+ptq5LzwbyrHZYq5FNk7ocufPv/ZQrcSSC+cFwJBAKvOJByS -x56qyGeZLOQlWS2JS3KJo59XuLFGqcbgN9Om9xFa41Yb4N9NvplFivsvZdw3m1Q/ -SPIXQuT8RMPDVNQ= +MIIG/wIBADANBgkqhkiG9w0BAQEFAASCBukwggblAgEAAoIBgQCylKlLaKU+hOvJ +DfriTRLd+IthG5hv28I3A/CGjLICT0rDDtgaXd0uqloJAnjsgn5gMAcStpDW8Rm+ +t6LsrBL+5fBgkyU1r94Rvx0HHoyaZwBBouitVHw28hP3W+smddkqB1UxpGnTeL2B +gj3dVo/WTtRfO+0h0PKw1l98YE1pMTdqIwcOOE/ER0g4hvA/wrxuLhMvlVLMy/lL +58uctqaDUqryNyeerKbVkq4fJyCG5D2TwXVJ3i2DDh0xSt2Y10poZV4M4k8Su9Z5 +8zN2PSvYMT50aqF277v8BaOeYUApBE4kZGIJpo13ATGdEwpUFZ0Fri4zLYUZ1hWb +OC35sKo7OxWQ/+tefNUdgWHob6Vmy777jiYcLwxc3sS9rF3AJe0rMW83kCkR6hmy +A3250E137N/1QumHuT/Nj9rnI/lwt9jfaYkZjoAgT/C97m/mM83cYpGTdoGV1xNo +7G90MhP0di5FnVsrIaSnvkbGT9UgUWx0oVMjocifdG2qIhMI9psCAwEAAQKCAYBT +sHmaPmNaZj59jZCqp0YVQlpHWwBYQ5vD3pPE6oCttm0p9nXt/VkfenQRTthOtmT1 +POzDp00/feP7zeGLmqSYUjgRekPw4gdnN7Ip2PY5kdW77NWwDSzdLxuOS8Rq1MW9 +/Yu+ZPe3RBlDbT8C0IM+Atlh/BqIQ3zIxN4g0pzUlF0M33d6AYfYSzOcUhibOO7H +j84r+YXBNkIRgYKZYbutRXuZYaGuqejRpBj3voVu0d3Ntdb6lCWuClpB9HzfGN0c +RTv8g6UYO4sK3qyFn90ibIR/1GB9watvtoWVZqggiWeBzSWVWRsGEf9O+Cx4oJw1 +IphglhmhbgNksbj7bD24on/icldSOiVkoUemUOFmHWhCm4PnB1GmbD8YMfEdSbks +qDr1Ps1zg4mGOinVD/4cY7vuPFO/HCH07wfeaUGzRt4g0/yLr+XjVofOA3oowyxv +JAzr+niHA3lg5ecj4r7M68efwzN1OCyjMrVJw2RAzwvGxE+rm5NiT08SWlKQZnkC +gcEA4wvyLpIur/UB84nV3XVJ89UMNBLm++aTFzld047BLJtMaOhvNqx6Cl5c8VuW +l261KHjiVzpfNM3/A2LBQJcYkhX7avkqEXlj57cl+dCWAVwUzKmLJTPjfaTTZnYJ +xeN3dMYjJz2z2WtgvfvDoJLukVwIMmhTY8wtqqYyQBJ/l06pBsfw5TNvmVIOQHds +8ASOiFt+WRLk2bl9xrGGayqt3VV93KVRzF27cpjOgEcG74F3c0ZW9snERN7vIYwB +JfrlAoHBAMlahPwMP2TYylG8OzHe7EiehTekSO26LGh0Cq3wTGXYsK/q8hQCzL14 +kWW638vpwXL6L9ntvrd7hjzWRO3vX/VxnYEA6f0bpqHq1tZi6lzix5CTUN5McpDg +QnjenSJNrNjS1zEF8WeY9iLEuDI/M/iUW4y9R6s3WpgQhPDXpSvd2g3gMGRUYhxQ +Xna8auiJeYFq0oNaOxvJj+VeOfJ3ZMJttd+Y7gTOYZcbg3SdRb/kdxYki0RMD2hF +4ZvjJ6CTfwKBwQDiMqiZFTJGQwYqp4vWEmAW+I4r4xkUpWatoI2Fk5eI5T9+1PLX +uYXsho56NxEU1UrOg4Cb/p+TcBc8PErkGqR0BkpxDMOInTOXSrQe6lxIBoECVXc3 +HTbrmiay0a5y5GfCgxPKqIJhfcToAceoVjovv0y7S4yoxGZKuUEe7E8JY2iqRNAO +yOvKCCICv/hcN235E44RF+2/rDlOltagNej5tY6rIFkaDdgOF4bD7f9O5eEni1Bg +litfoesDtQP/3rECgcEAkQfvQ7D6tIPmbqsbJBfCr6fmoqZllT4FIJN84b50+OL0 +mTGsfjdqC4tdhx3sdu7/VPbaIqm5NmX10bowWgWSY7MbVME4yQPyqSwC5NbIonEC +d6N0mzoLR0kQ+Ai4u+2g82gicgAq2oj1uSNi3WZi48jQjHYFulCbo246o1NgeFFK +77WshYe2R1ioQfQDOU1URKCR0uTaMHClgfu112yiGd12JAD+aF3TM0kxDXz+sXI5 +SKy311DFxECZeXRLpcC3AoHBAJkNMJWTyPYbeVu+CTQkec8Uun233EkXa2kUNZc/ +5DuXDaK+A3DMgYRufTKSPpDHGaCZ1SYPInX1Uoe2dgVjWssRL2uitR4ENabDoAOA +ICVYXYYNagqQu5wwirF0QeaMXo1fjhuuHQh8GsMdXZvYEaAITZ9/NG5x/oY08+8H +kr78SMBOPy3XQn964uKG+e3JwpOG14GKABdAlrHKFXNWchu/6dgcYXB87mrC/GhO +zNwzC+QhFTZoOomFoqMgFWujng== -----END PRIVATE KEY----- diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 1e0f14e..9effddd 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -29,7 +29,7 @@ try: except ImportError: thread = None -__all__ = ["Error", "TestFailed", "ResourceDenied", "import_module", +__all__ = ["Error", "TestFailed", "TestDidNotRun", "ResourceDenied", "import_module", "verbose", "use_resources", "max_memuse", "record_original_stdout", "get_original_stdout", "unload", "unlink", "rmtree", "forget", "is_resource_enabled", "requires", "requires_mac_ver", @@ -53,6 +53,9 @@ class Error(Exception): class TestFailed(Error): """Test failed.""" +class TestDidNotRun(Error): + """Test did not run any subtests.""" + class ResourceDenied(unittest.SkipTest): """Test skipped because it requested a disallowed resource. @@ -654,8 +657,12 @@ if have_unicode: unichr(0x20AC), ): try: - character.encode(sys.getfilesystemencoding())\ - .decode(sys.getfilesystemencoding()) + # In Windows, 'mbcs' is used, and encode() returns '?' + # for characters missing in the ANSI codepage + if character.encode(sys.getfilesystemencoding())\ + .decode(sys.getfilesystemencoding())\ + != character: + raise UnicodeError except UnicodeError: pass else: @@ -1151,6 +1158,9 @@ def transient_internet(resource_name, timeout=30.0, errnos=()): ('EHOSTUNREACH', 113), ('ENETUNREACH', 101), ('ETIMEDOUT', 110), + # socket.create_connection() fails randomly with + # EADDRNOTAVAIL on Travis CI. + ('EADDRNOTAVAIL', 99), ] default_gai_errnos = [ ('EAI_AGAIN', -3), @@ -1533,6 +1543,8 @@ def _run_suite(suite): runner = BasicTestRunner() result = runner.run(suite) + if not result.testsRun and not result.skipped: + raise TestDidNotRun if not result.wasSuccessful(): if len(result.errors) == 1 and not result.failures: err = result.errors[0][1] @@ -1671,6 +1683,14 @@ def run_doctest(module, verbosity=None): #======================================================================= # Threading support to prevent reporting refleaks when running regrtest.py -R +# Flag used by saved_test_environment of test.libregrtest.save_env, +# to check if a test modified the environment. The flag should be set to False +# before running a new test. +# +# For example, threading_cleanup() sets the flag is the function fails +# to cleanup threads. +environment_altered = False + # NOTE: we use thread._count() rather than threading.enumerate() (or the # moral equivalent thereof) because a threading.Thread object is still alive # until its __bootstrap() method has returned, even after it has been @@ -1714,6 +1734,43 @@ def reap_threads(func): threading_cleanup(*key) return decorator + +@contextlib.contextmanager +def wait_threads_exit(timeout=60.0): + """ + bpo-31234: Context manager to wait until all threads created in the with + statement exit. + + Use thread.count() to check if threads exited. Indirectly, wait until + threads exit the internal t_bootstrap() C function of the thread module. + + threading_setup() and threading_cleanup() are designed to emit a warning + if a test leaves running threads in the background. This context manager + is designed to cleanup threads started by the thread.start_new_thread() + which doesn't allow to wait for thread exit, whereas thread.Thread has a + join() method. + """ + old_count = thread._count() + try: + yield + finally: + start_time = time.time() + deadline = start_time + timeout + while True: + count = thread._count() + if count <= old_count: + break + if time.time() > deadline: + dt = time.time() - start_time + msg = ("wait_threads() failed to cleanup %s " + "threads after %.1f seconds " + "(count: %s, old count: %s)" + % (count - old_count, dt, count, old_count)) + raise AssertionError(msg) + time.sleep(0.010) + gc_collect() + + def reap_children(): """Use this function at the end of test_main() whenever sub-processes are started. This will help ensure that no extra children (zombies) @@ -2017,6 +2074,66 @@ def _crash_python(): _testcapi._read_null() +def fd_count(): + """Count the number of open file descriptors. + """ + if sys.platform.startswith(('linux', 'freebsd')): + try: + names = os.listdir("/proc/self/fd") + # Substract one because listdir() opens internally a file + # descriptor to list the content of the /proc/self/fd/ directory. + return len(names) - 1 + except OSError as exc: + if exc.errno != errno.ENOENT: + raise + + MAXFD = 256 + if hasattr(os, 'sysconf'): + try: + MAXFD = os.sysconf("SC_OPEN_MAX") + except OSError: + pass + + old_modes = None + if sys.platform == 'win32': + # bpo-25306, bpo-31009: Call CrtSetReportMode() to not kill the process + # on invalid file descriptor if Python is compiled in debug mode + try: + import msvcrt + msvcrt.CrtSetReportMode + except (AttributeError, ImportError): + # no msvcrt or a release build + pass + else: + old_modes = {} + for report_type in (msvcrt.CRT_WARN, + msvcrt.CRT_ERROR, + msvcrt.CRT_ASSERT): + old_modes[report_type] = msvcrt.CrtSetReportMode(report_type, 0) + + try: + count = 0 + for fd in range(MAXFD): + try: + # Prefer dup() over fstat(). fstat() can require input/output + # whereas dup() doesn't. + fd2 = os.dup(fd) + except OSError as e: + if e.errno != errno.EBADF: + raise + else: + os.close(fd2) + count += 1 + finally: + if old_modes is not None: + for report_type in (msvcrt.CRT_WARN, + msvcrt.CRT_ERROR, + msvcrt.CRT_ASSERT): + msvcrt.CrtSetReportMode(report_type, old_modes[report_type]) + + return count + + class SaveSignals: """ Save an restore signal handlers. diff --git a/Lib/test/talos-2019-0758.pem b/Lib/test/talos-2019-0758.pem new file mode 100644 index 0000000..13b95a7 --- /dev/null +++ b/Lib/test/talos-2019-0758.pem @@ -0,0 +1,22 @@ +-----BEGIN CERTIFICATE----- +MIIDqDCCApKgAwIBAgIBAjALBgkqhkiG9w0BAQswHzELMAkGA1UEBhMCVUsxEDAO +BgNVBAMTB2NvZHktY2EwHhcNMTgwNjE4MTgwMDU4WhcNMjgwNjE0MTgwMDU4WjA7 +MQswCQYDVQQGEwJVSzEsMCoGA1UEAxMjY29kZW5vbWljb24tdm0tMi50ZXN0Lmxh +bC5jaXNjby5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC63fGB +J80A9Av1GB0bptslKRIUtJm8EeEu34HkDWbL6AJY0P8WfDtlXjlPaLqFa6sqH6ES +V48prSm1ZUbDSVL8R6BYVYpOlK8/48xk4pGTgRzv69gf5SGtQLwHy8UPBKgjSZoD +5a5k5wJXGswhKFFNqyyxqCvWmMnJWxXTt2XDCiWc4g4YAWi4O4+6SeeHVAV9rV7C +1wxqjzKovVe2uZOHjKEzJbbIU6JBPb6TRfMdRdYOw98n1VXDcKVgdX2DuuqjCzHP +WhU4Tw050M9NaK3eXp4Mh69VuiKoBGOLSOcS8reqHIU46Reg0hqeL8LIL6OhFHIF +j7HR6V1X6F+BfRS/AgMBAAGjgdYwgdMwCQYDVR0TBAIwADAdBgNVHQ4EFgQUOktp +HQjxDXXUg8prleY9jeLKeQ4wTwYDVR0jBEgwRoAUx6zgPygZ0ZErF9sPC4+5e2Io +UU+hI6QhMB8xCzAJBgNVBAYTAlVLMRAwDgYDVQQDEwdjb2R5LWNhggkA1QEAuwb7 +2s0wCQYDVR0SBAIwADAuBgNVHREEJzAlgiNjb2Rlbm9taWNvbi12bS0yLnRlc3Qu +bGFsLmNpc2NvLmNvbTAOBgNVHQ8BAf8EBAMCBaAwCwYDVR0fBAQwAjAAMAsGCSqG +SIb3DQEBCwOCAQEAvqantx2yBlM11RoFiCfi+AfSblXPdrIrHvccepV4pYc/yO6p +t1f2dxHQb8rWH3i6cWag/EgIZx+HJQvo0rgPY1BFJsX1WnYf1/znZpkUBGbVmlJr +t/dW1gSkNS6sPsM0Q+7HPgEv8CPDNK5eo7vU2seE0iWOkxSyVUuiCEY9ZVGaLVit +p0C78nZ35Pdv4I+1cosmHl28+es1WI22rrnmdBpH8J1eY6WvUw2xuZHLeNVN0TzV +Q3qq53AaCWuLOD1AjESWuUCxMZTK9DPS4JKXTK8RLyDeqOvJGjsSWp3kL0y3GaQ+ +10T1rfkKJub2+m9A9duin1fn6tHc2wSvB7m3DA== +-----END CERTIFICATE----- diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 83f6f27..f39565a 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -4652,7 +4652,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=None, metavar=tuple()) def test_nargs_None_metavar_length1(self): - self.do_test_no_exception(nargs=None, metavar=("1")) + self.do_test_no_exception(nargs=None, metavar=("1",)) def test_nargs_None_metavar_length2(self): self.do_test_exception(nargs=None, metavar=("1", "2")) @@ -4669,7 +4669,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="?", metavar=tuple()) def test_nargs_optional_metavar_length1(self): - self.do_test_no_exception(nargs="?", metavar=("1")) + self.do_test_no_exception(nargs="?", metavar=("1",)) def test_nargs_optional_metavar_length2(self): self.do_test_exception(nargs="?", metavar=("1", "2")) @@ -4686,7 +4686,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="*", metavar=tuple()) def test_nargs_zeroormore_metavar_length1(self): - self.do_test_no_exception(nargs="*", metavar=("1")) + self.do_test_exception(nargs="*", metavar=("1",)) def test_nargs_zeroormore_metavar_length2(self): self.do_test_no_exception(nargs="*", metavar=("1", "2")) @@ -4703,7 +4703,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="+", metavar=tuple()) def test_nargs_oneormore_metavar_length1(self): - self.do_test_no_exception(nargs="+", metavar=("1")) + self.do_test_exception(nargs="+", metavar=("1",)) def test_nargs_oneormore_metavar_length2(self): self.do_test_no_exception(nargs="+", metavar=("1", "2")) @@ -4720,7 +4720,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_no_exception(nargs="...", metavar=tuple()) def test_nargs_remainder_metavar_length1(self): - self.do_test_no_exception(nargs="...", metavar=("1")) + self.do_test_no_exception(nargs="...", metavar=("1",)) def test_nargs_remainder_metavar_length2(self): self.do_test_no_exception(nargs="...", metavar=("1", "2")) @@ -4737,7 +4737,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs="A...", metavar=tuple()) def test_nargs_parser_metavar_length1(self): - self.do_test_no_exception(nargs="A...", metavar=("1")) + self.do_test_no_exception(nargs="A...", metavar=("1",)) def test_nargs_parser_metavar_length2(self): self.do_test_exception(nargs="A...", metavar=("1", "2")) @@ -4754,7 +4754,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=1, metavar=tuple()) def test_nargs_1_metavar_length1(self): - self.do_test_no_exception(nargs=1, metavar=("1")) + self.do_test_no_exception(nargs=1, metavar=("1",)) def test_nargs_1_metavar_length2(self): self.do_test_exception(nargs=1, metavar=("1", "2")) @@ -4771,7 +4771,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=2, metavar=tuple()) def test_nargs_2_metavar_length1(self): - self.do_test_no_exception(nargs=2, metavar=("1")) + self.do_test_exception(nargs=2, metavar=("1",)) def test_nargs_2_metavar_length2(self): self.do_test_no_exception(nargs=2, metavar=("1", "2")) @@ -4788,7 +4788,7 @@ class TestAddArgumentMetavar(TestCase): self.do_test_exception(nargs=3, metavar=tuple()) def test_nargs_3_metavar_length1(self): - self.do_test_no_exception(nargs=3, metavar=("1")) + self.do_test_exception(nargs=3, metavar=("1",)) def test_nargs_3_metavar_length2(self): self.do_test_exception(nargs=3, metavar=("1", "2")) @@ -4815,6 +4815,30 @@ class TestImportStar(TestCase): ] self.assertEqual(sorted(items), sorted(argparse.__all__)) + +class TestWrappingMetavar(TestCase): + + def setUp(self): + self.parser = ErrorRaisingArgumentParser( + 'this_is_spammy_prog_with_a_long_name_sorry_about_the_name' + ) + # this metavar was triggering library assertion errors due to usage + # message formatting incorrectly splitting on the ] chars within + metavar = '<http[s]://example:1234>' + self.parser.add_argument('--proxy', metavar=metavar) + + def test_help_with_metavar(self): + help_text = self.parser.format_help() + self.assertEqual(help_text, textwrap.dedent('''\ + usage: this_is_spammy_prog_with_a_long_name_sorry_about_the_name + [-h] [--proxy <http[s]://example:1234>] + + optional arguments: + -h, --help show this help message and exit + --proxy <http[s]://example:1234> + ''')) + + def test_main(): # silence warnings about version argument - these are expected with test_support.check_warnings( diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py index 693d67c..4b347a3 100644 --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -727,19 +727,20 @@ class BaseTestAPI(unittest.TestCase): server = TCPServer() t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() - self.addCleanup(t.join) - - for x in xrange(20): - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - s.settimeout(.2) - s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, - struct.pack('ii', 1, 0)) - try: - s.connect(server.address) - except socket.error: - pass - finally: - s.close() + try: + for x in xrange(20): + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.settimeout(.2) + s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, + struct.pack('ii', 1, 0)) + try: + s.connect(server.address) + except socket.error: + pass + finally: + s.close() + finally: + t.join() class TestAPI_UseSelect(BaseTestAPI): diff --git a/Lib/test/test_bdb.py b/Lib/test/test_bdb.py new file mode 100644 index 0000000..3ad70e2 --- /dev/null +++ b/Lib/test/test_bdb.py @@ -0,0 +1,1034 @@ +""" Test the bdb module. + + A test defines a list of tuples that may be seen as paired tuples, each + pair being defined by 'expect_tuple, set_tuple' as follows: + + ([event, [lineno[, co_name[, eargs]]]]), (set_type, [sargs]) + + * 'expect_tuple' describes the expected current state of the Bdb instance. + It may be the empty tuple and no check is done in that case. + * 'set_tuple' defines the set_*() method to be invoked when the Bdb + instance reaches this state. + + Example of an 'expect_tuple, set_tuple' pair: + + ('line', 2, 'tfunc_main'), ('step', ) + + Definitions of the members of the 'expect_tuple': + event: + Name of the trace event. The set methods that do not give back + control to the tracer [1] do not trigger a tracer event and in + that case the next 'event' may be 'None' by convention, its value + is not checked. + [1] Methods that trigger a trace event are set_step(), set_next(), + set_return(), set_until() and set_continue(). + lineno: + Line number. Line numbers are relative to the start of the + function when tracing a function in the test_bdb module (i.e. this + module). + co_name: + Name of the function being currently traced. + eargs: + A tuple: + * On an 'exception' event the tuple holds a class object, the + current exception must be an instance of this class. + * On a 'line' event, the tuple holds a dictionary and a list. The + dictionary maps each breakpoint number that has been hit on this + line to its hits count. The list holds the list of breakpoint + number temporaries that are being deleted. + + Definitions of the members of the 'set_tuple': + set_type: + The type of the set method to be invoked. This may + be the type of one of the Bdb set methods: 'step', 'next', + 'until', 'return', 'continue', 'break', 'quit', or the type of one + of the set methods added by test_bdb.Bdb: 'ignore', 'enable', + 'disable', 'clear', 'up', 'down'. + sargs: + The arguments of the set method if any, packed in a tuple. +""" + +from __future__ import absolute_import +from __future__ import print_function + +import bdb as _bdb +import sys +import os +import unittest +import textwrap +import importlib +import linecache +from contextlib import contextmanager +from itertools import islice, repeat +import test.support + +class BdbException(Exception): pass +class BdbError(BdbException): """Error raised by the Bdb instance.""" +class BdbSyntaxError(BdbException): """Syntax error in the test case.""" +class BdbNotExpectedError(BdbException): """Unexpected result.""" + +# When 'dry_run' is set to true, expect tuples are ignored and the actual +# state of the tracer is printed after running each set_*() method of the test +# case. The full list of breakpoints and their attributes is also printed +# after each 'line' event where a breakpoint has been hit. +dry_run = 0 + +__file__ = os.path.splitext(__file__)[0] + '.py' + +def reset_Breakpoint(): + _bdb.Breakpoint.next = 1 + _bdb.Breakpoint.bplist = {} + _bdb.Breakpoint.bpbynumber = [None] + +def info_breakpoints(): + bp_list = [bp for bp in _bdb.Breakpoint.bpbynumber if bp] + if not bp_list: + return '' + + header_added = False + for bp in bp_list: + if not header_added: + info = 'BpNum Temp Enb Hits Ignore Where\n' + header_added = True + + disp = 'yes ' if bp.temporary else 'no ' + enab = 'yes' if bp.enabled else 'no ' + info += ('%-5d %s %s %-4d %-6d at %s:%d' % + (bp.number, disp, enab, bp.hits, bp.ignore, + os.path.basename(bp.file), bp.line)) + if bp.cond: + info += '\n\tstop only if %s' % (bp.cond,) + info += '\n' + return info + +class Bdb(_bdb.Bdb, object): + """Extend Bdb to enhance test coverage.""" + + def trace_dispatch(self, frame, event, arg): + self.currentbp = None + return super(Bdb, self).trace_dispatch(frame, event, arg) + + def set_break(self, filename, lineno, temporary=False, cond=None, + funcname=None): + if isinstance(funcname, str): + if filename == __file__: + globals_ = globals() + else: + module = importlib.import_module(filename[:-3]) + globals_ = module.__dict__ + func = eval(funcname, globals_) + code = func.__code__ + filename = code.co_filename + lineno = code.co_firstlineno + funcname = code.co_name + + res = super(Bdb, self).set_break(filename, lineno, + temporary=temporary, cond=cond, funcname=funcname) + if isinstance(res, str): + raise BdbError(res) + return res + + # Back port of get_bpbynumber() from bdb.Bdb in Python 3. + def get_bpbynumber(self, arg): + """Return a breakpoint by its index in Breakpoint.bybpnumber. + + For invalid arg values or if the breakpoint doesn't exist, + raise a ValueError. + """ + if not arg: + raise ValueError('Breakpoint number expected') + try: + number = int(arg) + except ValueError: + raise ValueError('Non-numeric breakpoint number %s' % arg) + try: + bp = _bdb.Breakpoint.bpbynumber[number] + except IndexError: + raise ValueError('Breakpoint number %d out of range' % number) + if bp is None: + raise ValueError('Breakpoint %d already deleted' % number) + return bp + + def get_stack(self, f, t): + self.stack, self.index = super(Bdb, self).get_stack(f, t) + self.frame = self.stack[self.index][0] + return self.stack, self.index + + def set_ignore(self, bpnum): + """Increment the ignore count of Breakpoint number 'bpnum'.""" + bp = self.get_bpbynumber(bpnum) + bp.ignore += 1 + + def set_enable(self, bpnum): + bp = self.get_bpbynumber(bpnum) + bp.enabled = True + + def set_disable(self, bpnum): + bp = self.get_bpbynumber(bpnum) + bp.enabled = False + + def set_clear(self, fname, lineno): + err = self.clear_break(fname, lineno) + if err: + raise BdbError(err) + + def set_up(self): + """Move up in the frame stack.""" + if not self.index: + raise BdbError('Oldest frame') + self.index -= 1 + self.frame = self.stack[self.index][0] + + def set_down(self): + """Move down in the frame stack.""" + if self.index + 1 == len(self.stack): + raise BdbError('Newest frame') + self.index += 1 + self.frame = self.stack[self.index][0] + +class Tracer(Bdb): + """A tracer for testing the bdb module.""" + + def __init__(self, expect_set, skip=None, dry_run=False, test_case=None): + super(Tracer, self).__init__(skip=skip) + self.expect_set = expect_set + self.dry_run = dry_run + self.header = ('Dry-run results for %s:' % test_case if + test_case is not None else None) + self.init_test() + + def init_test(self): + self.cur_except = None + self.expect_set_no = 0 + self.breakpoint_hits = None + self.expected_list = list(islice(self.expect_set, 0, None, 2)) + self.set_list = list(islice(self.expect_set, 1, None, 2)) + + def trace_dispatch(self, frame, event, arg): + # On an 'exception' event, call_exc_trace() in Python/ceval.c discards + # a BdbException raised by the Tracer instance, so we raise it on the + # next trace_dispatch() call that occurs unless the set_quit() or + # set_continue() method has been invoked on the 'exception' event. + if self.cur_except is not None: + raise self.cur_except + + if event == 'exception': + try: + res = super(Tracer, self).trace_dispatch(frame, event, arg) + return res + except BdbException as e: + self.cur_except = e + return self.trace_dispatch + else: + return super(Tracer, self).trace_dispatch(frame, event, arg) + + def user_call(self, frame, argument_list): + # Adopt the same behavior as pdb and, as a side effect, skip also the + # first 'call' event when the Tracer is started with Tracer.runcall() + # which may be possibly considered as a bug. + if not self.stop_here(frame): + return + self.process_event('call', frame, argument_list) + self.next_set_method() + + def user_line(self, frame): + self.process_event('line', frame) + + if self.dry_run and self.breakpoint_hits: + info = info_breakpoints().strip('\n') + # Indent each line. + for line in info.split('\n'): + print(' ' + line) + self.delete_temporaries() + self.breakpoint_hits = None + + self.next_set_method() + + def user_return(self, frame, return_value): + self.process_event('return', frame, return_value) + self.next_set_method() + + def user_exception(self, frame, exc_info): + self.exc_info = exc_info + self.process_event('exception', frame) + self.next_set_method() + + def do_clear(self, arg): + # The temporary breakpoints are deleted in user_line(). + bp_list = [self.currentbp] + self.breakpoint_hits = (bp_list, bp_list) + + def delete_temporaries(self): + if self.breakpoint_hits: + for n in self.breakpoint_hits[1]: + self.clear_bpbynumber(n) + + def pop_next(self): + self.expect_set_no += 1 + try: + self.expect = self.expected_list.pop(0) + except IndexError: + raise BdbNotExpectedError( + 'expect_set list exhausted, cannot pop item %d' % + self.expect_set_no) + self.set_tuple = self.set_list.pop(0) + + def process_event(self, event, frame, *args): + # Call get_stack() to enable walking the stack with set_up() and + # set_down(). + tb = None + if event == 'exception': + tb = self.exc_info[2] + self.get_stack(frame, tb) + + # A breakpoint has been hit and it is not a temporary. + if self.currentbp is not None and not self.breakpoint_hits: + bp_list = [self.currentbp] + self.breakpoint_hits = (bp_list, []) + + # Pop next event. + self.event= event + self.pop_next() + if self.dry_run: + self.print_state(self.header) + return + + # Validate the expected results. + if self.expect: + self.check_equal(self.expect[0], event, 'Wrong event type') + self.check_lno_name() + + if event in ('call', 'return'): + self.check_expect_max_size(3) + elif len(self.expect) > 3: + if event == 'line': + bps, temporaries = self.expect[3] + bpnums = sorted(bps.keys()) + if not self.breakpoint_hits: + self.raise_not_expected( + 'No breakpoints hit at expect_set item %d' % + self.expect_set_no) + self.check_equal(bpnums, self.breakpoint_hits[0], + 'Breakpoint numbers do not match') + self.check_equal([bps[n] for n in bpnums], + [self.get_bpbynumber(n).hits for + n in self.breakpoint_hits[0]], + 'Wrong breakpoint hit count') + self.check_equal(sorted(temporaries), self.breakpoint_hits[1], + 'Wrong temporary breakpoints') + + elif event == 'exception': + if not isinstance(self.exc_info[1], self.expect[3]): + self.raise_not_expected( + "Wrong exception at expect_set item %d, got '%s'" % + (self.expect_set_no, self.exc_info)) + + def check_equal(self, expected, result, msg): + if expected == result: + return + self.raise_not_expected("%s at expect_set item %d, got '%s'" % + (msg, self.expect_set_no, result)) + + def check_lno_name(self): + """Check the line number and function co_name.""" + s = len(self.expect) + if s > 1: + lineno = self.lno_abs2rel() + self.check_equal(self.expect[1], lineno, 'Wrong line number') + if s > 2: + self.check_equal(self.expect[2], self.frame.f_code.co_name, + 'Wrong function name') + + def check_expect_max_size(self, size): + if len(self.expect) > size: + raise BdbSyntaxError('Invalid size of the %s expect tuple: %s' % + (self.event, self.expect)) + + def lno_abs2rel(self): + fname = self.canonic(self.frame.f_code.co_filename) + lineno = self.frame.f_lineno + return ((lineno - self.frame.f_code.co_firstlineno + 1) + if fname == self.canonic(__file__) else lineno) + + def lno_rel2abs(self, fname, lineno): + return (self.frame.f_code.co_firstlineno + lineno - 1 + if (lineno and self.canonic(fname) == self.canonic(__file__)) + else lineno) + + def get_state(self): + lineno = self.lno_abs2rel() + co_name = self.frame.f_code.co_name + state = "('%s', %d, '%s'" % (self.event, lineno, co_name) + if self.breakpoint_hits: + bps = '{' + for n in self.breakpoint_hits[0]: + if bps != '{': + bps += ', ' + bps += '%s: %s' % (n, self.get_bpbynumber(n).hits) + bps += '}' + bps = '(' + bps + ', ' + str(self.breakpoint_hits[1]) + ')' + state += ', ' + bps + elif self.event == 'exception': + state += ', ' + self.exc_info[0].__name__ + state += '), ' + return state.ljust(32) + str(self.set_tuple) + ',' + + def print_state(self, header=None): + if header is not None and self.expect_set_no == 1: + print() + print(header) + print('%d: %s' % (self.expect_set_no, self.get_state())) + + def raise_not_expected(self, msg): + msg += '\n' + msg += ' Expected: %s\n' % str(self.expect) + msg += ' Got: ' + self.get_state() + raise BdbNotExpectedError(msg) + + def next_set_method(self): + set_type = self.set_tuple[0] + args = self.set_tuple[1] if len(self.set_tuple) == 2 else None + set_method = getattr(self, 'set_' + set_type) + + # The following set methods give back control to the tracer. + if set_type in ('step', 'continue', 'quit'): + set_method() + return + elif set_type in ('next', 'return'): + set_method(self.frame) + return + elif set_type == 'until': + set_method(self.frame) + return + + # The following set methods do not give back control to the tracer and + # next_set_method() is called recursively. + if (args and set_type in ('break', 'clear', 'ignore', 'enable', + 'disable')) or set_type in ('up', 'down'): + if set_type in ('break', 'clear'): + fname = args[0] + lineno = args[1] + lineno = self.lno_rel2abs(fname, lineno) + args = [fname, lineno] + list(args[2:]) + set_method(*args) + elif set_type in ('ignore', 'enable', 'disable'): + set_method(*args) + elif set_type in ('up', 'down'): + set_method() + + # Process the next expect_set item. + # It is not expected that a test may reach the recursion limit. + self.event= None + self.pop_next() + if self.dry_run: + self.print_state() + else: + if self.expect: + self.check_lno_name() + self.check_expect_max_size(3) + self.next_set_method() + else: + raise BdbSyntaxError('"%s" is an invalid set_tuple' % + self.set_tuple) + +class TracerRun(): + """Provide a context for running a Tracer instance with a test case.""" + + def __init__(self, test_case, skip=None): + self.test_case = test_case + self.dry_run = test_case.dry_run + self.tracer = Tracer(test_case.expect_set, skip=skip, + dry_run=self.dry_run, test_case=test_case.id()) + + def __enter__(self): + # test_pdb does not reset Breakpoint class attributes on exit :-( + reset_Breakpoint() + return self.tracer + + def __exit__(self, type_=None, value=None, traceback=None): + reset_Breakpoint() + sys.settrace(None) + + not_empty = '' + if self.tracer.set_list: + not_empty += 'All paired tuples have not been processed, ' + not_empty += ('the last one was number %d' % + self.tracer.expect_set_no) + + # Make a BdbNotExpectedError a unittest failure. + if type_ is not None and issubclass(BdbNotExpectedError, type_): + if isinstance(value, BaseException) and value.args: + err_msg = value.args[0] + if not_empty: + err_msg += '\n' + not_empty + if self.dry_run: + print(err_msg) + return True + else: + self.test_case.fail(err_msg) + else: + assert False, 'BdbNotExpectedError with empty args' + + if not_empty: + if self.dry_run: + print(not_empty) + else: + self.test_case.fail(not_empty) + +def run_test(modules, set_list, skip=None): + """Run a test and print the dry-run results. + + 'modules': A dictionary mapping module names to their source code as a + string. The dictionary MUST include one module named + 'test_module' with a main() function. + 'set_list': A list of set_type tuples to be run on the module. + + For example, running the following script outputs the following results: + + ***************************** SCRIPT ******************************** + + from test.test_bdb import run_test, break_in_func + + code = ''' + def func(): + lno = 3 + + def main(): + func() + lno = 7 + ''' + + set_list = [ + break_in_func('func', 'test_module.py'), + ('continue', ), + ('step', ), + ('step', ), + ('step', ), + ('quit', ), + ] + + modules = { 'test_module': code } + run_test(modules, set_list) + + **************************** results ******************************** + + 1: ('line', 2, 'tfunc_import'), ('next',), + 2: ('line', 3, 'tfunc_import'), ('step',), + 3: ('call', 5, 'main'), ('break', ('test_module.py', None, False, None, 'func')), + 4: ('None', 5, 'main'), ('continue',), + 5: ('line', 3, 'func', ({1: 1}, [])), ('step',), + BpNum Temp Enb Hits Ignore Where + 1 no yes 1 0 at test_module.py:2 + 6: ('return', 3, 'func'), ('step',), + 7: ('line', 7, 'main'), ('step',), + 8: ('return', 7, 'main'), ('quit',), + + ************************************************************************* + + """ + def gen(a, b): + try: + while 1: + x = next(a) + y = next(b) + yield x + yield y + except StopIteration: + return + + # Step over the import statement in tfunc_import using 'next' and step + # into main() in test_module. + sl = [('next', ), ('step', )] + sl.extend(set_list) + + test = BaseTestCase() + test.dry_run = True + test.id = lambda : None + test.expect_set = list(gen(repeat(()), iter(sl))) + with create_modules(modules): + sys.path.append(os.getcwd()) + with TracerRun(test, skip=skip) as tracer: + tracer.runcall(tfunc_import) + +@contextmanager +def create_modules(modules): + with test.support.temp_cwd(): + try: + sys.path.insert(0, os.getcwd()) + for m in modules: + fname = m + '.py' + with open(fname, 'w') as f: + f.write(textwrap.dedent(modules[m])) + linecache.checkcache(fname) + yield + finally: + sys.path.pop(0) + for m in modules: + test.support.forget(m) + +def break_in_func(funcname, fname=__file__, temporary=False, cond=None): + return 'break', (fname, None, temporary, cond, funcname) + +TEST_MODULE = 'test_module_for_bdb' +TEST_MODULE_FNAME = TEST_MODULE + '.py' +def tfunc_import(): + import test_module_for_bdb + test_module_for_bdb.main() + +def tfunc_main(): + lno = 2 + tfunc_first() + tfunc_second() + lno = 5 + lno = 6 + lno = 7 + +def tfunc_first(): + lno = 2 + lno = 3 + lno = 4 + +def tfunc_second(): + lno = 2 + +class BaseTestCase(unittest.TestCase): + """Base class for all tests.""" + dry_run = dry_run + +class StateTestCase(BaseTestCase): + """Test the step, next, return, until and quit 'set_' methods.""" + + def test_step(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_step_on_last_statement(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('break', (__file__, 3)), + ('None', 1, 'tfunc_first'), ('continue', ), + ('line', 3, 'tfunc_first', ({1:1}, [])), ('step', ), + ('line', 4, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next_on_last_statement(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('break', (__file__, 3)), + ('None', 1, 'tfunc_first'), ('continue', ), + ('line', 3, 'tfunc_first', ({1:1}, [])), ('next', ), + ('line', 4, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('next', ), + ('line', 4, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_second'), ('step', ), + ('line', 2, 'tfunc_second'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next_over_import(self): + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('next', ), + ('line', 3, 'tfunc_import'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_next_on_plain_statement(self): + # Check that set_next() is equivalent to set_step() on a plain + # statement. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('next', ), + ('line', 2, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_next_in_caller_frame(self): + # Check that set_next() in the caller frame causes the tracer + # to stop next in the caller frame. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('next', ), + ('line', 4, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_return(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('return', ), + ('return', 4, 'tfunc_first'), ('step', ), + ('line', 4, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_return_in_caller_frame(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('return', ), + ('return', 7, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_until(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('step', ), + ('line', 2, 'tfunc_first'), ('until', ), + ('line', 3, 'tfunc_first'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_until_in_caller_frame(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('until', ), + ('line', 4, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + + def test_skip(self): + # Check that tracing is skipped over the import statement in + # 'tfunc_import()'. + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('step', ), + ('line', 3, 'tfunc_import'), ('quit', ), + ] + skip = ('importlib*', TEST_MODULE) + with TracerRun(self, skip=skip) as tracer: + tracer.runcall(tfunc_import) + + def test_down(self): + # Check that set_down() raises BdbError at the newest frame. + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('down', ), + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_main) + + def test_up(self): + self.expect_set = [ + ('line', 2, 'tfunc_main'), ('step', ), + ('line', 3, 'tfunc_main'), ('step', ), + ('call', 1, 'tfunc_first'), ('up', ), + ('None', 3, 'tfunc_main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_main) + +class BreakpointTestCase(BaseTestCase): + """Test the breakpoint set method.""" + + def test_bp_on_non_existent_module(self): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', ('/non/existent/module.py', 1)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + + def test_bp_after_last_statement(self): + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 4)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + + def test_temporary_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(2): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [1])), ('continue', ), + ('line', 3, 'func', ({2:1}, [2])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_disabled_temporary_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, True), + ('None', 2, 'tfunc_import'), ('disable', (2, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('enable', (2, )), + ('None', 3, 'func'), ('disable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({2:1}, [2])), ('enable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_condition(self): + code = """ + def func(a): + lno = 3 + + def main(): + for i in range(3): + func(i) + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, False, 'a == 2'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:3}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_exception_on_condition_evaluation(self): + code = """ + def func(a): + lno = 3 + + def main(): + func(0) + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME, False, '1 // 0'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_bp_ignore_count(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(2): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('ignore', (1, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_ignore_count_on_disabled_bp(self): + code = """ + def func(): + lno = 3 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), + break_in_func('func', TEST_MODULE_FNAME), + ('None', 2, 'tfunc_import'), ('ignore', (1, )), + ('None', 2, 'tfunc_import'), ('disable', (1, )), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({2:1}, [])), ('enable', (1, )), + ('None', 3, 'func'), ('continue', ), + ('line', 3, 'func', ({2:2}, [])), ('continue', ), + ('line', 3, 'func', ({1:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_clear_two_bp_on_same_line(self): + code = """ + def func(): + lno = 3 + lno = 4 + + def main(): + for i in range(3): + func() + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 3)), + ('None', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 3)), + ('None', 2, 'tfunc_import'), ('break', (TEST_MODULE_FNAME, 4)), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('continue', ), + ('line', 4, 'func', ({3:1}, [])), ('clear', (TEST_MODULE_FNAME, 3)), + ('None', 4, 'func'), ('continue', ), + ('line', 4, 'func', ({3:2}, [])), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + + def test_clear_at_no_bp(self): + self.expect_set = [ + ('line', 2, 'tfunc_import'), ('clear', (__file__, 1)) + ] + with TracerRun(self) as tracer: + self.assertRaises(BdbError, tracer.runcall, tfunc_import) + +class RunTestCase(BaseTestCase): + """Test run, runeval and set_trace.""" + + def test_run_step(self): + # Check that the bdb 'run' method stops at the first line event. + code = """ + lno = 2 + """ + self.expect_set = [ + ('line', 2, '<module>'), ('step', ), + ('return', 2, '<module>'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.run(compile(textwrap.dedent(code), '<string>', 'exec')) + + def test_runeval_step(self): + # Test bdb 'runeval'. + code = """ + def main(): + lno = 3 + """ + modules = { TEST_MODULE: code } + with create_modules(modules): + self.expect_set = [ + ('line', 1, '<module>'), ('step', ), + ('call', 2, 'main'), ('step', ), + ('line', 3, 'main'), ('step', ), + ('return', 3, 'main'), ('step', ), + ('return', 1, '<module>'), ('quit', ), + ] + import test_module_for_bdb + with TracerRun(self) as tracer: + tracer.runeval('test_module_for_bdb.main()', globals(), locals()) + +class IssuesTestCase(BaseTestCase): + """Test fixed bdb issues.""" + + def test_step_at_return_with_no_trace_in_caller(self): + # Issue #13183. + # Check that the tracer does step into the caller frame when the + # trace function is not set in that frame. + code_1 = """ + from test_module_for_bdb_2 import func + def main(): + func() + lno = 5 + """ + code_2 = """ + def func(): + lno = 3 + """ + modules = { + TEST_MODULE: code_1, + 'test_module_for_bdb_2': code_2, + } + with create_modules(modules): + self.expect_set = [ + ('line', 2, 'tfunc_import'), + break_in_func('func', 'test_module_for_bdb_2.py'), + ('None', 2, 'tfunc_import'), ('continue', ), + ('line', 3, 'func', ({1:1}, [])), ('step', ), + ('return', 3, 'func'), ('step', ), + ('line', 5, 'main'), ('quit', ), + ] + with TracerRun(self) as tracer: + tracer.runcall(tfunc_import) + +def test_main(): + test.support.run_unittest( + StateTestCase, + RunTestCase, + BreakpointTestCase, + IssuesTestCase, + ) + +if __name__ == "__main__": + test_main() diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index ce2c5b2..cb2a4d9 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -4,6 +4,7 @@ XXX This is a mess. Common tests should be unified with string_tests.py (and the latter should be modernized). """ +import array import os import re import sys @@ -58,11 +59,49 @@ class BaseBytesTest(unittest.TestCase): self.assertRaises(IndexError, lambda: b[-sys.maxint-2]) self.assertRaises(IndexError, lambda: b[-10**100]) + def test_from_iterable(self): + b = self.type2test(range(256)) + self.assertEqual(len(b), 256) + self.assertEqual(list(b), list(range(256))) + + # Non-sequence iterable. + b = self.type2test({42}) + self.assertEqual(b, b"*") + b = self.type2test({43, 45}) + self.assertIn(tuple(b), {(43, 45), (45, 43)}) + + # Iterator that has a __length_hint__. + b = self.type2test(iter(range(256))) + self.assertEqual(len(b), 256) + self.assertEqual(list(b), list(range(256))) + + # Iterator that doesn't have a __length_hint__. + b = self.type2test(i for i in range(256) if i % 2) + self.assertEqual(len(b), 128) + self.assertEqual(list(b), list(range(256))[1::2]) + + # Sequence without __iter__. + class S: + def __getitem__(self, i): + return (1, 2, 3)[i] + b = self.type2test(S()) + self.assertEqual(b, b"\x01\x02\x03") + + def test_from_tuple(self): + # There is a special case for tuples. + b = self.type2test(tuple(range(256))) + self.assertEqual(len(b), 256) + self.assertEqual(list(b), list(range(256))) + b = self.type2test((1, 2, 3)) + self.assertEqual(b, b"\x01\x02\x03") + def test_from_list(self): - ints = list(range(256)) - b = self.type2test(i for i in ints) + # There is a special case for lists. + b = self.type2test(list(range(256))) self.assertEqual(len(b), 256) - self.assertEqual(list(b), ints) + self.assertEqual(list(b), list(range(256))) + b = self.type2test([1, 2, 3]) + self.assertEqual(b, b"\x01\x02\x03") def test_from_index(self): b = self.type2test([Indexable(), Indexable(1), Indexable(254), @@ -71,6 +110,20 @@ class BaseBytesTest(unittest.TestCase): self.assertRaises(ValueError, self.type2test, [Indexable(-1)]) self.assertRaises(ValueError, self.type2test, [Indexable(256)]) + def test_from_buffer(self): + a = self.type2test(array.array('B', [1, 2, 3])) + self.assertEqual(a, b"\x01\x02\x03") + a = self.type2test(b"\x01\x02\x03") + self.assertEqual(a, b"\x01\x02\x03") + + # Issues #29159 and #34974. + # Fallback when __index__ raises a TypeError + class B(bytes): + def __index__(self): + raise TypeError + + self.assertEqual(self.type2test(B(b"foobar")), b"foobar") + def test_from_ssize(self): self.assertEqual(self.type2test(0), b'') self.assertEqual(self.type2test(1), b'\x00') @@ -103,6 +156,20 @@ class BaseBytesTest(unittest.TestCase): self.assertRaises(ValueError, self.type2test, [sys.maxint+1]) self.assertRaises(ValueError, self.type2test, [10**100]) + def test_constructor_exceptions(self): + # Issue #34974: bytes and bytearray constructors replace unexpected + # exceptions. + class BadInt: + def __index__(self): + 1//0 + self.assertRaises(ZeroDivisionError, self.type2test, BadInt()) + self.assertRaises(ZeroDivisionError, self.type2test, [BadInt()]) + + class BadIterable: + def __iter__(self): + 1//0 + self.assertRaises(ZeroDivisionError, self.type2test, BadIterable()) + def test_compare(self): b1 = self.type2test([1, 2, 3]) b2 = self.type2test([1, 2, 3]) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 944e960..5eb3f7d 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -2,6 +2,7 @@ # these are all functions _testcapi exports whose name begins with 'test_'. from __future__ import with_statement +import string import sys import time import random @@ -101,6 +102,20 @@ class TestPendingCalls(unittest.TestCase): self.pendingcalls_wait(l, n) +class TestGetIndices(unittest.TestCase): + + def test_get_indices(self): + self.assertEqual(_testcapi.get_indices(slice(10L, 20, 1), 100), (0, 10, 20, 1)) + self.assertEqual(_testcapi.get_indices(slice(10.1, 20, 1), 100), None) + self.assertEqual(_testcapi.get_indices(slice(10, 20L, 1), 100), (0, 10, 20, 1)) + self.assertEqual(_testcapi.get_indices(slice(10, 20.1, 1), 100), None) + + self.assertEqual(_testcapi.get_indices(slice(10L, 20, 1L), 100), (0, 10, 20, 1)) + self.assertEqual(_testcapi.get_indices(slice(10.1, 20, 1L), 100), None) + self.assertEqual(_testcapi.get_indices(slice(10, 20L, 1L), 100), (0, 10, 20, 1)) + self.assertEqual(_testcapi.get_indices(slice(10, 20.1, 1L), 100), None) + + @unittest.skipUnless(threading and thread, 'Threading required for this test.') class TestThreadState(unittest.TestCase): @@ -126,18 +141,15 @@ class TestThreadState(unittest.TestCase): t.join() +class Test_testcapi(unittest.TestCase): + locals().update((name, getattr(_testcapi, name)) + for name in dir(_testcapi) + if name.startswith('test_') and not name.endswith('_code')) + + def test_main(): - for name in dir(_testcapi): - if name.startswith('test_'): - test = getattr(_testcapi, name) - if support.verbose: - print "internal", name - try: - test() - except _testcapi.error: - raise support.TestFailed, sys.exc_info()[1] - - support.run_unittest(CAPITest, TestPendingCalls, TestThreadState) + support.run_unittest(CAPITest, TestPendingCalls, + TestThreadState, TestGetIndices, Test_testcapi) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py index c9cf095..743c2af 100644 --- a/Lib/test/test_cgi.py +++ b/Lib/test/test_cgi.py @@ -1,3 +1,4 @@ +from io import BytesIO from test.test_support import run_unittest, check_warnings import cgi import os @@ -316,6 +317,60 @@ Content-Type: text/plain v = gen_result(data, environ) self.assertEqual(self._qs_result, v) + def test_max_num_fields(self): + # For application/x-www-form-urlencoded + data = '&'.join(['a=a']*11) + environ = { + 'CONTENT_LENGTH': str(len(data)), + 'CONTENT_TYPE': 'application/x-www-form-urlencoded', + 'REQUEST_METHOD': 'POST', + } + + with self.assertRaises(ValueError): + cgi.FieldStorage( + fp=BytesIO(data.encode()), + environ=environ, + max_num_fields=10, + ) + + # For multipart/form-data + data = """---123 +Content-Disposition: form-data; name="a" + +3 +---123 +Content-Type: application/x-www-form-urlencoded + +a=4 +---123 +Content-Type: application/x-www-form-urlencoded + +a=5 +---123-- +""" + environ = { + 'CONTENT_LENGTH': str(len(data)), + 'CONTENT_TYPE': 'multipart/form-data; boundary=-123', + 'QUERY_STRING': 'a=1&a=2', + 'REQUEST_METHOD': 'POST', + } + + # 2 GET entities + # 1 top level POST entities + # 1 entity within the second POST entity + # 1 entity within the third POST entity + with self.assertRaises(ValueError): + cgi.FieldStorage( + fp=BytesIO(data.encode()), + environ=environ, + max_num_fields=4, + ) + cgi.FieldStorage( + fp=BytesIO(data.encode()), + environ=environ, + max_num_fields=5, + ) + def testQSAndFormData(self): data = """ ---123 diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index e954a0c..15a00f3 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -392,7 +392,7 @@ if 1: 'from sys import stdin)', 'from sys import stdin, stdout,\nstderr', 'from sys import stdin si', - 'from sys import stdin,' + 'from sys import stdin,', 'from sys import (*)', 'from sys import (stdin,, stdout, stderr)', 'from sys import (stdin, stdout),', diff --git a/Lib/test/test_cprofile.py b/Lib/test/test_cprofile.py index af3fe62..83d5229 100644 --- a/Lib/test/test_cprofile.py +++ b/Lib/test/test_cprofile.py @@ -1,7 +1,9 @@ """Test suite for the cProfile module.""" import sys +import unittest from test.test_support import run_unittest, TESTFN, unlink +from test.support.script_helper import assert_python_failure # rip off all interesting stuff from test_profile import cProfile @@ -26,8 +28,14 @@ class CProfileTest(ProfileTest): unlink(TESTFN) +class TestCommandLine(unittest.TestCase): + def test_sort(self): + rc, out, err = assert_python_failure('-m', 'cProfile', '-s', 'demo') + self.assertGreater(rc, 0) + self.assertIn(b"option -s: invalid choice: 'demo'", err) + def test_main(): - run_unittest(CProfileTest) + run_unittest(CProfileTest, TestCommandLine) def main(): if '-r' not in sys.argv: diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py index f56c3f5..2620fbb 100644 --- a/Lib/test/test_datetime.py +++ b/Lib/test/test_datetime.py @@ -495,41 +495,64 @@ class TestTimeDelta(HarmlessMixedComparison, unittest.TestCase): def test_issue31752(self): # The interpreter shouldn't crash because divmod() returns negative # remainder. - class BadInt(int): - def __mul__(self, other): - return Prod() - - class BadLong(long): - def __mul__(self, other): - return Prod() - - class Prod: - def __radd__(self, other): - return Sum() - - class Sum(int): - def __divmod__(self, other): - # negative remainder - return (0, -1) - - timedelta(microseconds=BadInt(1)) - timedelta(hours=BadInt(1)) - timedelta(weeks=BadInt(1)) - timedelta(microseconds=BadLong(1)) - timedelta(hours=BadLong(1)) - timedelta(weeks=BadLong(1)) - - class Sum(long): - def __divmod__(self, other): - # negative remainder - return (0, -1) - - timedelta(microseconds=BadInt(1)) - timedelta(hours=BadInt(1)) - timedelta(weeks=BadInt(1)) - timedelta(microseconds=BadLong(1)) - timedelta(hours=BadLong(1)) - timedelta(weeks=BadLong(1)) + for inttype in (int, long): + class BadInt(inttype): + def __mul__(self, other): + return Prod() + def __rmul__(self, other): + return Prod() + def __floordiv__(self, other): + return Prod() + def __rfloordiv__(self, other): + return Prod() + + class BadLong(long): + def __mul__(self, other): + return Prod() + def __rmul__(self, other): + return Prod() + def __floordiv__(self, other): + return Prod() + def __rfloordiv__(self, other): + return Prod() + + class Prod: + def __add__(self, other): + return Sum() + def __radd__(self, other): + return Sum() + + for inttype2 in (int, long): + class Sum(inttype2): + def __divmod__(self, other): + return divmodresult + + for divmodresult in [None, (), (0, 1, 2), (0, -1)]: + # The following examples should not crash. + try: + timedelta(microseconds=BadInt(1)) + except TypeError: + pass + try: + timedelta(hours=BadInt(1)) + except TypeError: + pass + try: + timedelta(weeks=BadInt(1)) + except (TypeError, ValueError): + pass + try: + timedelta(1) * BadInt(1) + except (TypeError, ValueError): + pass + try: + BadInt(1) * timedelta(1) + except TypeError: + pass + try: + timedelta(1) // BadInt(1) + except TypeError: + pass ############################################################################# diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 14b7f42..f481075 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -914,10 +914,10 @@ class DecimalFormatTest(unittest.TestCase): decimal_point = locale.localeconv()['decimal_point'] thousands_sep = locale.localeconv()['thousands_sep'] if decimal_point != '\xd9\xab': - self.skipTest('inappropriate decimal point separator' + self.skipTest('inappropriate decimal point separator ' '({!r} not {!r})'.format(decimal_point, '\xd9\xab')) if thousands_sep != '\xd9\xac': - self.skipTest('inappropriate thousands separator' + self.skipTest('inappropriate thousands separator ' '({!r} not {!r})'.format(thousands_sep, '\xd9\xac')) self.assertEqual(format(Decimal('100000000.123'), 'n'), diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index c81064d..e6307ab 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -659,6 +659,21 @@ class TestSubclass(unittest.TestCase): d1 == d2 # not clear if this is supposed to be True or False, # but it used to give a SystemError + @test_support.cpython_only + def test_bug_31608(self): + # The interpreter used to crash in specific cases where a deque + # subclass returned a non-deque. + class X(deque): + pass + d = X() + def bad___new__(cls, *args, **kwargs): + return [42] + X.__new__ = bad___new__ + with self.assertRaises(TypeError): + d * 42 # shouldn't crash + with self.assertRaises(TypeError): + d + deque([1, 2, 3]) # shouldn't crash + class SubclassWithKwargs(deque): def __init__(self, newarg=1): diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 354c33b..a7a6845 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -2355,7 +2355,11 @@ def test_unittest_reportflags(): Then the default eporting options are ignored: >>> result = suite.run(unittest.TestResult()) - >>> print result.failures[0][1] # doctest: +ELLIPSIS + + *NOTE*: These doctest are intentionally not placed in raw string to depict + the trailing whitespace using `\x20` in the diff below. + + >>> print(result.failures[0][1]) # doctest: +ELLIPSIS Traceback ... Failed example: favorite_color @@ -2368,7 +2372,7 @@ def test_unittest_reportflags(): Differences (ndiff with -expected +actual): a - <BLANKLINE> - + + +\x20 b <BLANKLINE> <BLANKLINE> @@ -2717,6 +2721,47 @@ def old_test4(): """ TestResults(failed=0, attempted=4) """ +def test_no_trailing_whitespace_stripping(): + r""" + The fancy reports had a bug for a long time where any trailing whitespace on + the reported diff lines was stripped, making it impossible to see the + differences in line reported as different that differed only in the amount of + trailing whitespace. The whitespace still isn't particularly visible unless + you use NDIFF, but at least it is now there to be found. + + *NOTE*: This snippet was intentionally put inside a raw string to get rid of + leading whitespace error in executing the example below + + >>> def f(x): + ... r''' + ... >>> print('\n'.join(['a ', 'b'])) + ... a + ... b + ... ''' + """ + """ + *NOTE*: These doctest are not placed in raw string to depict the trailing whitespace + using `\x20` + + >>> test = doctest.DocTestFinder().find(f)[0] + >>> flags = doctest.REPORT_NDIFF + >>> doctest.DocTestRunner(verbose=False, optionflags=flags).run(test) + ... # doctest: +ELLIPSIS + ********************************************************************** + File ..., line 3, in f + Failed example: + print('\n'.join(['a ', 'b'])) + Differences (ndiff with -expected +actual): + - a + + a + b + TestResults(failed=1, attempted=1) + + *NOTE*: `\x20` is for checking the trailing whitespace on the +a line above. + We cannot use actual spaces there, as a commit hook prevents from committing + patches that contain trailing whitespace. More info on Issue 24746. + """ + ###################################################################### ## Main ###################################################################### diff --git a/Lib/test/test_file.py b/Lib/test/test_file.py index 1fd7910..6bfc4e6 100644 --- a/Lib/test/test_file.py +++ b/Lib/test/test_file.py @@ -12,7 +12,8 @@ from weakref import proxy import io import _pyio as pyio -from test.test_support import TESTFN, run_unittest +from test.support import TESTFN, run_unittest +from test import support from UserList import UserList class AutoFileTests(unittest.TestCase): @@ -24,7 +25,7 @@ class AutoFileTests(unittest.TestCase): def tearDown(self): if self.f: self.f.close() - os.remove(TESTFN) + support.unlink(TESTFN) def testWeakRefs(self): # verify weak references @@ -143,8 +144,12 @@ class PyAutoFileTests(AutoFileTests): class OtherFileTests(unittest.TestCase): + def tearDown(self): + support.unlink(TESTFN) + def testModeStrings(self): # check invalid mode strings + self.open(TESTFN, 'wb').close() for mode in ("", "aU", "wU+"): try: f = self.open(TESTFN, mode) @@ -191,7 +196,6 @@ class OtherFileTests(unittest.TestCase): # SF bug <http://www.python.org/sf/801631> # "file.truncate fault on windows" - os.unlink(TESTFN) f = self.open(TESTFN, 'wb') try: @@ -215,7 +219,6 @@ class OtherFileTests(unittest.TestCase): self.fail("File size after ftruncate wrong %d" % size) finally: f.close() - os.unlink(TESTFN) def testIteration(self): # Test the complex interaction when mixing file-iteration and the @@ -236,86 +239,82 @@ class OtherFileTests(unittest.TestCase): methods = [("readline", ()), ("read", ()), ("readlines", ()), ("readinto", (array("b", b" "*100),))] - try: - # Prepare the testfile - bag = self.open(TESTFN, "wb") - bag.write(filler * nchunks) - bag.writelines(testlines) - bag.close() - # Test for appropriate errors mixing read* and iteration - for methodname, args in methods: - f = self.open(TESTFN, 'rb') - if next(f) != filler: - self.fail, "Broken testfile" - meth = getattr(f, methodname) - meth(*args) # This simply shouldn't fail - f.close() - - # Test to see if harmless (by accident) mixing of read* and - # iteration still works. This depends on the size of the internal - # iteration buffer (currently 8192,) but we can test it in a - # flexible manner. Each line in the bag o' ham is 4 bytes - # ("h", "a", "m", "\n"), so 4096 lines of that should get us - # exactly on the buffer boundary for any power-of-2 buffersize - # between 4 and 16384 (inclusive). + # Prepare the testfile + bag = self.open(TESTFN, "wb") + bag.write(filler * nchunks) + bag.writelines(testlines) + bag.close() + # Test for appropriate errors mixing read* and iteration + for methodname, args in methods: f = self.open(TESTFN, 'rb') - for i in range(nchunks): - next(f) - testline = testlines.pop(0) - try: - line = f.readline() - except ValueError: - self.fail("readline() after next() with supposedly empty " - "iteration-buffer failed anyway") - if line != testline: - self.fail("readline() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - testline = testlines.pop(0) - buf = array("b", b"\x00" * len(testline)) + self.assertEqual(next(f), filler) + meth = getattr(f, methodname) + meth(*args) # This simply shouldn't fail + f.close() + + # Test to see if harmless (by accident) mixing of read* and + # iteration still works. This depends on the size of the internal + # iteration buffer (currently 8192,) but we can test it in a + # flexible manner. Each line in the bag o' ham is 4 bytes + # ("h", "a", "m", "\n"), so 4096 lines of that should get us + # exactly on the buffer boundary for any power-of-2 buffersize + # between 4 and 16384 (inclusive). + f = self.open(TESTFN, 'rb') + for i in range(nchunks): + next(f) + testline = testlines.pop(0) + try: + line = f.readline() + except ValueError: + self.fail("readline() after next() with supposedly empty " + "iteration-buffer failed anyway") + if line != testline: + self.fail("readline() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + testline = testlines.pop(0) + buf = array("b", b"\x00" * len(testline)) + try: + f.readinto(buf) + except ValueError: + self.fail("readinto() after next() with supposedly empty " + "iteration-buffer failed anyway") + line = buf.tostring() + if line != testline: + self.fail("readinto() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + + testline = testlines.pop(0) + try: + line = f.read(len(testline)) + except ValueError: + self.fail("read() after next() with supposedly empty " + "iteration-buffer failed anyway") + if line != testline: + self.fail("read() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + try: + lines = f.readlines() + except ValueError: + self.fail("readlines() after next() with supposedly empty " + "iteration-buffer failed anyway") + if lines != testlines: + self.fail("readlines() after next() with empty buffer " + "failed. Got %r, expected %r" % (line, testline)) + # Reading after iteration hit EOF shouldn't hurt either + f.close() + f = self.open(TESTFN, 'rb') + try: + for line in f: + pass try: + f.readline() f.readinto(buf) + f.read() + f.readlines() except ValueError: - self.fail("readinto() after next() with supposedly empty " - "iteration-buffer failed anyway") - line = buf.tostring() - if line != testline: - self.fail("readinto() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - - testline = testlines.pop(0) - try: - line = f.read(len(testline)) - except ValueError: - self.fail("read() after next() with supposedly empty " - "iteration-buffer failed anyway") - if line != testline: - self.fail("read() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - try: - lines = f.readlines() - except ValueError: - self.fail("readlines() after next() with supposedly empty " - "iteration-buffer failed anyway") - if lines != testlines: - self.fail("readlines() after next() with empty buffer " - "failed. Got %r, expected %r" % (line, testline)) - # Reading after iteration hit EOF shouldn't hurt either - f.close() - f = self.open(TESTFN, 'rb') - try: - for line in f: - pass - try: - f.readline() - f.readinto(buf) - f.read() - f.readlines() - except ValueError: - self.fail("read* failed after next() consumed file") - finally: - f.close() + self.fail("read* failed after next() consumed file") finally: - os.unlink(TESTFN) + f.close() class COtherFileTests(OtherFileTests): open = io.open @@ -325,14 +324,8 @@ class PyOtherFileTests(OtherFileTests): def test_main(): - # Historically, these tests have been sloppy about removing TESTFN. - # So get rid of it no matter what. - try: - run_unittest(CAutoFileTests, PyAutoFileTests, - COtherFileTests, PyOtherFileTests) - finally: - if os.path.exists(TESTFN): - os.unlink(TESTFN) + run_unittest(CAutoFileTests, PyAutoFileTests, + COtherFileTests, PyOtherFileTests) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py index 2825a87..57c9f14 100644 --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -12,7 +12,7 @@ from functools import wraps from UserList import UserList from test.test_support import TESTFN, check_warnings, run_unittest, make_bad_fd -from test.test_support import py3k_bytes as bytes, cpython_only +from test.test_support import py3k_bytes as bytes, cpython_only, check_py3k_warnings from test.script_helper import run_python from _io import FileIO as _FileIO @@ -101,6 +101,10 @@ class AutoFileTests(unittest.TestCase): self.assertEqual(self.f.readline(None), b"hi\n") self.assertEqual(self.f.readlines(None), [b"bye\n", b"abc"]) + def testWriteUnicode(self): + with check_py3k_warnings(): + self.f.write(u'') + def testRepr(self): self.assertEqual(repr(self.f), "<_io.FileIO name=%r mode='%s'>" % (self.f.name, self.f.mode)) @@ -210,7 +214,7 @@ class AutoFileTests(unittest.TestCase): @ClosedFDRaises def testErrnoOnClosedWrite(self, f): - f.write('a') + f.write(b'a') @ClosedFDRaises def testErrnoOnClosedSeek(self, f): diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py index e728aa7..8a3eb06 100644 --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -675,6 +675,7 @@ class TestTLS_FTPClass(TestCase): # clear text sock = self.client.transfercmd('list') self.assertNotIsInstance(sock, ssl.SSLSocket) + self.assertEqual(sock.recv(1024), LIST_DATA.encode('ascii')) sock.close() self.assertEqual(self.client.voidresp(), "226 transfer complete") @@ -682,6 +683,9 @@ class TestTLS_FTPClass(TestCase): self.client.prot_p() sock = self.client.transfercmd('list') self.assertIsInstance(sock, ssl.SSLSocket) + # consume from SSL socket to finalize handshake and avoid + # "SSLError [SSL] shutdown while in init" + self.assertEqual(sock.recv(1024), LIST_DATA.encode('ascii')) sock.close() self.assertEqual(self.client.voidresp(), "226 transfer complete") @@ -689,6 +693,7 @@ class TestTLS_FTPClass(TestCase): self.client.prot_c() sock = self.client.transfercmd('list') self.assertNotIsInstance(sock, ssl.SSLSocket) + self.assertEqual(sock.recv(1024), LIST_DATA.encode('ascii')) sock.close() self.assertEqual(self.client.voidresp(), "226 transfer complete") diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index dce3c5c..335e48a 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -3,13 +3,14 @@ # The code for testing gdb was adapted from similar work in Unladen Swallow's # Lib/test/test_jit_gdb.py +import locale import os import re import subprocess import sys import sysconfig +import textwrap import unittest -import sysconfig from test import test_support from test.test_support import run_unittest, findfile @@ -57,6 +58,23 @@ if sys.platform.startswith("sunos"): checkout_hook_path = os.path.join(os.path.dirname(sys.executable), 'python-gdb.py') + +def cet_protection(): + cflags = sysconfig.get_config_var('CFLAGS') + if not cflags: + return False + flags = cflags.split() + # True if "-mcet -fcf-protection" options are found, but false + # if "-fcf-protection=none" or "-fcf-protection=return" is found. + return (('-mcet' in flags) + and any((flag.startswith('-fcf-protection') + and not flag.endswith(("=none", "=return"))) + for flag in flags)) + +# Control-flow enforcement technology +CET_PROTECTION = cet_protection() + + def run_gdb(*args, **env_vars): """Runs gdb in batch mode with the additional arguments given by *args. @@ -170,6 +188,12 @@ class DebuggerTests(unittest.TestCase): commands += ['set print entry-values no'] if cmds_after_breakpoint: + if CET_PROTECTION: + # bpo-32962: When Python is compiled with -mcet + # -fcf-protection, function arguments are unusable before + # running the first instruction of the function entry point. + # The 'next' command makes the required first step. + commands += ['next'] commands += cmds_after_breakpoint else: commands += ['backtrace'] @@ -214,6 +238,15 @@ class DebuggerTests(unittest.TestCase): for line in errlines: if not line: continue + # bpo34007: Sometimes some versions of the shared libraries that + # are part of the traceback are compiled in optimised mode and the + # Program Counter (PC) is not present, not allowing gdb to walk the + # frames back. When this happens, the Python bindings of gdb raise + # an exception, making the test impossible to succeed. + if "PC not saved" in line: + raise unittest.SkipTest("gdb cannot walk the frame object" + " because the Program Counter is" + " not present") if not line.startswith(ignore_patterns): unexpected_errlines.append(line) @@ -231,6 +264,11 @@ class DebuggerTests(unittest.TestCase): # # For a nested structure, the first time we hit the breakpoint will # give us the top-level structure + + # NOTE: avoid decoding too much of the traceback as some + # undecodable characters may lurk there in optimized mode + # (issue #19743). + cmds_after_breakpoint = cmds_after_breakpoint or ["backtrace 1"] gdb_output = self.get_stack_trace(source, breakpoint='PyObject_Print', cmds_after_breakpoint=cmds_after_breakpoint, import_site=import_site) @@ -385,7 +423,7 @@ except RuntimeError, e: # Test division by zero: gdb_repr, gdb_output = self.get_gdb_repr(''' try: - a = 1 / 0 + a = 1 // 0 except ZeroDivisionError, e: print e ''') @@ -863,7 +901,32 @@ print 42 breakpoint='time_gmtime', cmds_after_breakpoint=['py-bt-full'], ) - self.assertIn('#0 <built-in function gmtime', gdb_output) + self.assertIn('#1 <built-in function gmtime', gdb_output) + + @unittest.skipIf(python_is_optimized(), + "Python was compiled with optimizations") + def test_wrapper_call(self): + cmd = textwrap.dedent(''' + class MyList(list): + def __init__(self): + super(MyList, self).__init__() # wrapper_call() + + print("first break point") + l = MyList() + ''') + cmds_after_breakpoint = ['break wrapper_call', 'continue'] + if CET_PROTECTION: + # bpo-32962: same case as in get_stack_trace(): + # we need an additional 'next' command in order to read + # arguments of the innermost function of the call stack. + cmds_after_breakpoint.append('next') + cmds_after_breakpoint.append('py-bt') + + # Verify with "py-bt": + gdb_output = self.get_stack_trace(cmd, + cmds_after_breakpoint=cmds_after_breakpoint) + self.assertRegexpMatches(gdb_output, + r"<method-wrapper u?'__init__' of MyList object at ") class PyPrintTests(DebuggerTests): diff --git a/Lib/test/test_gdbm.py b/Lib/test/test_gdbm.py index e9169a2..c4a498f 100644 --- a/Lib/test/test_gdbm.py +++ b/Lib/test/test_gdbm.py @@ -62,9 +62,13 @@ class TestGdbm(unittest.TestCase): self.g = gdbm.open(filename, 'c') size0 = os.path.getsize(filename) - self.g['x'] = 'x' * 10000 + # bpo-33901: on macOS with gdbm 1.15, an empty database uses 16 MiB + # and adding an entry of 10,000 B has no effect on the file size. + # Add size0 bytes to make sure that the file size changes. + value_size = max(size0, 10000) + self.g['x'] = 'x' * value_size size1 = os.path.getsize(filename) - self.assertTrue(size0 < size1) + self.assertGreater(size1, size0) del self.g['x'] # 'size' is supposed to be the same even after deleting an entry. @@ -72,7 +76,8 @@ class TestGdbm(unittest.TestCase): self.g.reorganize() size2 = os.path.getsize(filename) - self.assertTrue(size1 > size2 >= size0) + self.assertLess(size2, size1) + self.assertGreaterEqual(size2, size0) def test_main(): diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index 0f7bf19..10285f6 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -1877,6 +1877,16 @@ test_generators just happened to be the test that drew these out. """ +crash_test = """ +>>> def foo(): yield +>>> gen = foo() +>>> gen.next() +>>> print gen.gi_frame.f_restricted # This would segfault. +False + +""" + + __test__ = {"tut": tutorial_tests, "pep": pep_tests, "email": email_tests, @@ -1886,6 +1896,7 @@ __test__ = {"tut": tutorial_tests, "weakref": weakref_tests, "coroutine": coroutine_tests, "refleaks": refleaks_tests, + "crash": crash_test, } # Magic test name that regrtest.py invokes *after* importing this module. diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py index e0efcaa..7f819fb 100644 --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -1,9 +1,10 @@ import unittest import math +import string import sys from test import test_support # Skip this test if the _testcapi module isn't available. -test_support.import_module('_testcapi') +_testcapi = test_support.import_module('_testcapi') from _testcapi import getargs_keywords import warnings @@ -799,12 +800,6 @@ class Unicode_TestCase(unittest.TestCase): self.assertRaises(TypeError, getargs_u_hash, None) -def test_main(): - tests = [Signed_TestCase, Unsigned_TestCase, LongLong_TestCase, - Tuple_TestCase, Keywords_TestCase, - Bytes_TestCase, String_TestCase, Unicode_TestCase] - test_support.run_unittest(*tests) - class Object_TestCase(unittest.TestCase): def test_S(self): from _testcapi import getargs_S @@ -840,5 +835,142 @@ class Object_TestCase(unittest.TestCase): self.assertRaises(TypeError, getargs_U, buffer(obj)) +class SkipitemTest(unittest.TestCase): + + def test_skipitem(self): + """ + If this test failed, you probably added a new "format unit" + in Python/getargs.c, but neglected to update our poor friend + skipitem() in the same file. (If so, shame on you!) + + With a few exceptions**, this function brute-force tests all + printable ASCII*** characters (32 to 126 inclusive) as format units, + checking to see that PyArg_ParseTupleAndKeywords() return consistent + errors both when the unit is attempted to be used and when it is + skipped. If the format unit doesn't exist, we'll get one of two + specific error messages (one for used, one for skipped); if it does + exist we *won't* get that error--we'll get either no error or some + other error. If we get the specific "does not exist" error for one + test and not for the other, there's a mismatch, and the test fails. + + ** Some format units have special funny semantics and it would + be difficult to accommodate them here. Since these are all + well-established and properly skipped in skipitem() we can + get away with not testing them--this test is really intended + to catch *new* format units. + + *** Python C source files must be ASCII. Therefore it's impossible + to have non-ASCII format units. + + """ + empty_tuple = () + tuple_1 = (0,) + dict_b = {'b':1} + keywords = ["a", "b"] + + for i in range(32, 127): + c = chr(i) + + # skip parentheses, the error reporting is inconsistent about them + # skip 'e', it's always a two-character code + # skip '|', it doesn't represent arguments anyway + if c in '()e|': + continue + + # test the format unit when not skipped + format = c + "i" + try: + _testcapi.parse_tuple_and_keywords(tuple_1, dict_b, + format, keywords) + when_not_skipped = False + except TypeError as e: + s = "argument 1 (impossible<bad format char>)" + when_not_skipped = (str(e) == s) + except RuntimeError: + when_not_skipped = False + + # test the format unit when skipped + optional_format = "|" + format + try: + _testcapi.parse_tuple_and_keywords(empty_tuple, dict_b, + optional_format, keywords) + when_skipped = False + except RuntimeError as e: + s = "impossible<bad format char>: '{}'".format(format) + when_skipped = (str(e) == s) + + message = ("test_skipitem_parity: " + "detected mismatch between convertsimple and skipitem " + "for format unit '{}' ({}), not skipped {}, skipped {}".format( + c, i, when_skipped, when_not_skipped)) + self.assertIs(when_skipped, when_not_skipped, message) + + def test_skipitem_with_suffix(self): + parse = _testcapi.parse_tuple_and_keywords + empty_tuple = () + tuple_1 = (0,) + dict_b = {'b':1} + keywords = ["a", "b"] + + supported = ('s#', 's*', 'z#', 'z*', 'u#', 't#', 'w#', 'w*') + for c in string.ascii_letters: + for c2 in '#*': + f = c + c2 + optional_format = "|" + f + "i" + if f in supported: + parse(empty_tuple, dict_b, optional_format, keywords) + else: + with self.assertRaisesRegexp((RuntimeError, TypeError), + 'impossible<bad format char>'): + parse(empty_tuple, dict_b, optional_format, keywords) + + for c in map(chr, range(32, 128)): + f = 'e' + c + optional_format = "|" + f + "i" + if c in 'st': + parse(empty_tuple, dict_b, optional_format, keywords) + else: + with self.assertRaisesRegexp(RuntimeError, + 'impossible<bad format char>'): + parse(empty_tuple, dict_b, optional_format, keywords) + + +class ParseTupleAndKeywords_Test(unittest.TestCase): + + def test_parse_tuple_and_keywords(self): + # Test handling errors in the parse_tuple_and_keywords helper itself + self.assertRaises(TypeError, _testcapi.parse_tuple_and_keywords, + (), {}, 42, []) + self.assertRaises(ValueError, _testcapi.parse_tuple_and_keywords, + (), {}, '', 42) + self.assertRaises(ValueError, _testcapi.parse_tuple_and_keywords, + (), {}, '', [''] * 42) + self.assertRaises(TypeError, _testcapi.parse_tuple_and_keywords, + (), {}, '', [42]) + + def test_bad_use(self): + # Test handling invalid format and keywords in + # PyArg_ParseTupleAndKeywords() + self.assertRaises(TypeError, _testcapi.parse_tuple_and_keywords, + (1,), {}, '||O', ['a']) + self.assertRaises(RuntimeError, _testcapi.parse_tuple_and_keywords, + (1,), {}, '|O', ['a', 'b']) + self.assertRaises(RuntimeError, _testcapi.parse_tuple_and_keywords, + (1,), {}, '|OO', ['a']) + + +class Test_testcapi(unittest.TestCase): + locals().update((name, getattr(_testcapi, name)) + for name in dir(_testcapi) + if name.startswith('test_') and name.endswith('_code')) + + +def test_main(): + tests = [Signed_TestCase, Unsigned_TestCase, LongLong_TestCase, + Tuple_TestCase, Keywords_TestCase, + Bytes_TestCase, String_TestCase, Unicode_TestCase, + SkipitemTest, ParseTupleAndKeywords_Test, Test_testcapi] + test_support.run_unittest(*tests) + if __name__ == "__main__": test_main() diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index 23b6ce8..228586e 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -5,6 +5,7 @@ from test.test_support import run_unittest, check_syntax_error, \ check_py3k_warnings import unittest import sys +import warnings # testing import * from sys import * @@ -628,7 +629,6 @@ hello world with check_py3k_warnings((warntext, DeprecationWarning)): compile(code, '<test string>', 'exec') if sys.py3kwarning: - import warnings with warnings.catch_warnings(): warnings.filterwarnings('error', category=DeprecationWarning) with self.assertRaises(SyntaxError) as cm: @@ -883,6 +883,13 @@ hello world with check_py3k_warnings(('<> not supported in 3.x; use !=', DeprecationWarning)): if eval('1 < 1 > 1 == 1 >= 1 <= 1 <> 1 != 1 in 1 not in 1 is 1 is not 1'): pass + if sys.py3kwarning: + with warnings.catch_warnings(): + warnings.filterwarnings('error', category=DeprecationWarning) + with self.assertRaises(DeprecationWarning) as cm: + compile('1 <> 1', '<test string>', 'eval') + self.assertIn('<> not supported in 3.x; use !=', + str(cm.exception)) def test_binary_mask_ops(self): x = 1 & 1 diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py index 902d93f..cdb1af5 100644 --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -6,6 +6,7 @@ from test import test_support import os import io import struct +import tempfile gzip = test_support.import_module('gzip') data1 = """ int length=DEFAULTALLOC, err = Z_OK; @@ -331,6 +332,12 @@ class TestGzip(unittest.TestCase): with gzip.GzipFile(fileobj=f, mode="w") as g: self.assertEqual(g.name, "") + def test_fileobj_from_io_open(self): + fd = os.open(self.filename, os.O_WRONLY | os.O_CREAT) + with io.open(fd, "wb") as f: + with gzip.GzipFile(fileobj=f, mode="w") as g: + self.assertEqual(g.name, "") + def test_fileobj_mode(self): gzip.GzipFile(self.filename, "wb").close() with open(self.filename, "r+b") as f: @@ -359,6 +366,14 @@ class TestGzip(unittest.TestCase): with gzip.GzipFile(fileobj=io.BytesIO(gzdata)) as f: self.assertEqual(f.read(), b'Test') + def test_fileobj_without_name(self): + # Issue #33038: GzipFile should not assume that file objects that have + # a .name attribute use a non-None value. + with tempfile.SpooledTemporaryFile() as f: + with gzip.GzipFile(fileobj=f, mode='wb') as archive: + archive.write(b'data') + self.assertEqual(archive.name, '') + def test_main(verbose=None): test_support.run_unittest(TestGzip) diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py index 471ebb4..b8d6388 100644 --- a/Lib/test/test_hashlib.py +++ b/Lib/test/test_hashlib.py @@ -371,25 +371,25 @@ class HashLibTestCase(unittest.TestCase): data = smallest_data*200000 expected_hash = hashlib.sha1(data*num_threads).hexdigest() - def hash_in_chunks(chunk_size, event): + def hash_in_chunks(chunk_size): index = 0 while index < len(data): hasher.update(data[index:index+chunk_size]) index += chunk_size - event.set() - events = [] + threads = [] for threadnum in xrange(num_threads): chunk_size = len(data) // (10**threadnum) assert chunk_size > 0 assert chunk_size % len(smallest_data) == 0 - event = threading.Event() - events.append(event) - threading.Thread(target=hash_in_chunks, - args=(chunk_size, event)).start() - - for event in events: - event.wait() + thread = threading.Thread(target=hash_in_chunks, + args=(chunk_size,)) + threads.append(thread) + + for thread in threads: + thread.start() + for thread in threads: + thread.join() self.assertEqual(expected_hash, hasher.hexdigest()) diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py index 11f0d5d..25236c6 100644 --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -55,8 +55,8 @@ class TestServerThread(threading.Thread): self.test_object = test_object def run(self): - self.server = HTTPServer(('', 0), self.request_handler) - self.test_object.PORT = self.server.socket.getsockname()[1] + self.server = HTTPServer(('localhost', 0), self.request_handler) + self.test_object.HOST, self.test_object.PORT = self.server.socket.getsockname() self.test_object.server_started.set() self.test_object = None try: @@ -66,6 +66,7 @@ class TestServerThread(threading.Thread): def stop(self): self.server.shutdown() + self.join() class BaseTestCase(unittest.TestCase): @@ -83,7 +84,7 @@ class BaseTestCase(unittest.TestCase): test_support.threading_cleanup(*self._threads) def request(self, uri, method='GET', body=None, headers={}): - self.connection = httplib.HTTPConnection('localhost', self.PORT) + self.connection = httplib.HTTPConnection(self.HOST, self.PORT) self.connection.request(method, uri, body, headers) return self.connection.getresponse() @@ -186,7 +187,7 @@ class BaseHTTPServerTestCase(BaseTestCase): def setUp(self): BaseTestCase.setUp(self) - self.con = httplib.HTTPConnection('localhost', self.PORT) + self.con = httplib.HTTPConnection(self.HOST, self.PORT) self.con.connect() def test_command(self): diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py index 405b7ea..acaad63 100644 --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -166,14 +166,18 @@ class BaseThreadedNetworkedTests(unittest.TestCase): def test_linetoolong(self): + maxline = 10 + class TooLongHandler(SimpleIMAPHandler): def handle(self): # Send a very long response line - self.wfile.write('* OK ' + imaplib._MAXLINE*'x' + '\r\n') + self.wfile.write('* OK ' + maxline * 'x' + '\r\n') - with self.reaped_server(TooLongHandler) as server: - self.assertRaises(imaplib.IMAP4.error, - self.imap_class, *server.server_address) + with self.reaped_server(TooLongHandler) as server, \ + support.swap_attr(imaplib, '_MAXLINE', maxline): + with self.assertRaisesRegexp(imaplib.IMAP4.error, + 'got more than 10 bytes'): + self.imap_class(*server.server_address) class ThreadedNetworkedTests(BaseThreadedNetworkedTests): @@ -187,9 +191,6 @@ class ThreadedNetworkedTestsSSL(BaseThreadedNetworkedTests): server_class = SecureTCPServer imap_class = IMAP4_SSL - def test_linetoolong(self): - raise unittest.SkipTest("test is not reliable on 2.7; see issue 20118") - class RemoteIMAPTest(unittest.TestCase): host = 'cyrus.andrew.cmu.edu' diff --git a/Lib/test/test_import_magic.py b/Lib/test/test_import_magic.py index 92f68b4..57e350e 100644 --- a/Lib/test/test_import_magic.py +++ b/Lib/test/test_import_magic.py @@ -11,7 +11,7 @@ class MagicNumberTests(unittest.TestCase): Test release compatibility issues relating to precompiled bytecode """ @unittest.skipUnless( - sys.version_info.releaselevel in ('final', 'release'), + sys.version_info.releaselevel in ('candidate', 'final'), 'only applies to candidate or final python release levels' ) def test_magic_number(self): diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 7d09c6f..5063a30 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -209,7 +209,7 @@ class GetSourceBase(unittest.TestCase): def sourcerange(self, top, bottom): lines = self.source.split("\n") - return "\n".join(lines[top-1:bottom]) + "\n" + return "\n".join(lines[top-1:bottom]) + ("\n" if bottom else "") def assertSourceEqual(self, obj, top, bottom): self.assertEqual(inspect.getsource(obj), @@ -331,6 +331,16 @@ class TestRetrievingSourceCode(GetSourceBase): finally: linecache.getlines = getlines +class TestGettingSourceOfToplevelFrames(GetSourceBase): + fodderFile = mod + + def test_range_toplevel_frame(self): + self.maxDiff = None + self.assertSourceEqual(mod.currentframe, 1, None) + + def test_range_traceback_toplevel_frame(self): + self.assertSourceEqual(mod.tb, 1, None) + class TestDecorators(GetSourceBase): fodderFile = mod2 @@ -896,7 +906,8 @@ def test_main(): TestDecorators, TestRetrievingSourceCode, TestOneliners, TestBuggyCases, TestInterpreterStack, TestClassesAndFunctions, TestPredicates, TestGetcallargsFunctions, TestGetcallargsFunctionsCellVars, - TestGetcallargsMethods, TestGetcallargsUnboundMethods) + TestGetcallargsMethods, TestGetcallargsUnboundMethods, + TestGettingSourceOfToplevelFrames) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index ea5ec65..5ec0b7b 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -690,6 +690,16 @@ class IOTest(unittest.TestCase): self.assertEqual(stream.readinto(buffer), 5) self.assertEqual(buffer.tobytes(), b"12345") + def test_close_assert(self): + class R(self.IOBase): + def __setattr__(self, name, value): + pass + def flush(self): + raise OSError() + f = R() + # This would cause an assertion failure. + self.assertRaises(OSError, f.close) + class CIOTest(IOTest): @@ -1091,6 +1101,7 @@ class CBufferedReaderTest(BufferedReaderTest, SizeofTest): def test_garbage_collection(self): # C BufferedReader objects are collected. # The Python version has __del__, so it ends into gc.garbage instead + self.addCleanup(support.unlink, support.TESTFN) rawio = self.FileIO(support.TESTFN, "w+b") f = self.tp(rawio) f.f = f @@ -1292,6 +1303,7 @@ class BufferedWriterTest(unittest.TestCase, CommonBufferedTests): def test_truncate(self): # Truncate implicitly flushes the buffer. + self.addCleanup(support.unlink, support.TESTFN) with self.open(support.TESTFN, self.write_mode, buffering=0) as raw: bufio = self.tp(raw, 8) bufio.write(b"abcdef") @@ -1398,6 +1410,7 @@ class CBufferedWriterTest(BufferedWriterTest, SizeofTest): # C BufferedWriter objects are collected, and collecting them flushes # all data to disk. # The Python version has __del__, so it ends into gc.garbage instead + self.addCleanup(support.unlink, support.TESTFN) rawio = self.FileIO(support.TESTFN, "w+b") f = self.tp(rawio) f.write(b"123xxx") @@ -2738,6 +2751,17 @@ class TextIOWrapperTest(unittest.TestCase): with self.maybeRaises(TypeError): t.read(42) + def test_issue25862(self): + # Assertion failures occurred in tell() after read() and write(). + t = self.TextIOWrapper(self.BytesIO(b'test'), encoding='ascii') + t.read(1) + t.read() + t.tell() + t = self.TextIOWrapper(self.BytesIO(b'test'), encoding='ascii') + t.read(1) + t.write('x') + t.tell() + class CTextIOWrapperTest(TextIOWrapperTest): @@ -2783,6 +2807,11 @@ class CTextIOWrapperTest(TextIOWrapperTest): t2.buddy = t1 support.gc_collect() + def test_del__CHUNK_SIZE_SystemError(self): + t = self.TextIOWrapper(self.BytesIO(), encoding='ascii') + with self.assertRaises(AttributeError): + del t._CHUNK_SIZE + maybeRaises = unittest.TestCase.assertRaises @@ -2898,6 +2927,16 @@ class IncrementalNewlineDecoderTest(unittest.TestCase): dec = self.IncrementalNewlineDecoder(None, translate=True) _check(dec) + def test_translate(self): + # issue 35062 + for translate in (-2, -1, 1, 2): + decoder = codecs.getincrementaldecoder("utf-8")() + decoder = self.IncrementalNewlineDecoder(decoder, translate) + self.check_newline_decoding_utf8(decoder) + decoder = codecs.getincrementaldecoder("utf-8")() + decoder = self.IncrementalNewlineDecoder(decoder, translate=0) + self.assertEqual(decoder.decode(b"\r\r\n"), "\r\r\n") + class CIncrementalNewlineDecoderTest(IncrementalNewlineDecoderTest): pass @@ -3146,7 +3185,6 @@ class SignalsTest(unittest.TestCase): try: wio = self.io.open(w, **fdopen_kwargs) t.start() - signal.alarm(1) # Fill the pipe enough that the write will be blocking. # It will be interrupted by the timer armed above. Since the # other thread has read one byte, the low-level write will @@ -3154,10 +3192,13 @@ class SignalsTest(unittest.TestCase): # The buffered IO layer must check for pending signal # handlers, which in this case will invoke alarm_interrupt(). try: + signal.alarm(1) with self.assertRaises(ZeroDivisionError): wio.write(item * (support.PIPE_MAX_SIZE // len(item) + 1)) finally: + signal.alarm(0) t.join() + # We got one byte, get another one and check that it isn't a # repeat of the first one. read_results.append(os.read(r, 1)) @@ -3206,6 +3247,7 @@ class SignalsTest(unittest.TestCase): if isinstance(exc, RuntimeError): self.assertTrue(str(exc).startswith("reentrant call"), str(exc)) finally: + signal.alarm(0) wio.close() os.close(r) @@ -3234,6 +3276,7 @@ class SignalsTest(unittest.TestCase): # - third raw read() returns b"bar" self.assertEqual(decode(rio.read(6)), "foobar") finally: + signal.alarm(0) rio.close() os.close(w) os.close(r) @@ -3295,6 +3338,7 @@ class SignalsTest(unittest.TestCase): self.assertIsNone(error[0]) self.assertEqual(N, sum(len(x) for x in read_results)) finally: + signal.alarm(0) write_finished = True os.close(w) os.close(r) diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 563ddb1..6070882 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -425,7 +425,7 @@ class NormalizeTest(unittest.TestCase): def test_valencia_modifier(self): self.check('ca_ES.UTF-8@valencia', 'ca_ES.UTF-8@valencia') - self.check('ca_ES@valencia', 'ca_ES.ISO8859-15@valencia') + self.check('ca_ES@valencia', 'ca_ES.UTF-8@valencia') self.check('ca@valencia', 'ca_ES.ISO8859-1@valencia') def test_devanagari_modifier(self): diff --git a/Lib/test/test_marshal.py b/Lib/test/test_marshal.py index 76d59ba..bbaa944 100644 --- a/Lib/test/test_marshal.py +++ b/Lib/test/test_marshal.py @@ -227,8 +227,22 @@ class BugsTestCase(unittest.TestCase): pass def test_loads_recursion(self): - s = 'c' + ('X' * 4*4) + '{' * 2**20 - self.assertRaises(ValueError, marshal.loads, s) + def run_tests(N, check): + # (((...None...),),) + check(b'(\x01\x00\x00\x00' * N + b'N') + # [[[...None...]]] + check(b'[\x01\x00\x00\x00' * N + b'N') + # {None: {None: {None: ...None...}}} + check(b'{N' * N + b'N' + b'0' * N) + # frozenset([frozenset([frozenset([...None...])])]) + check(b'>\x01\x00\x00\x00' * N + b'N') + # Check that the generated marshal data is valid and marshal.loads() + # works for moderately deep nesting + run_tests(100, marshal.loads) + # Very deeply nested structure shouldn't blow the stack + def check(s): + self.assertRaises(ValueError, marshal.loads, s) + run_tests(2**20, check) def test_recursion_limit(self): # Create a deeply nested structure. diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py index b6d88d2..2eb6423 100644 --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -3,7 +3,7 @@ import copy import pickle from StringIO import StringIO -from test.test_support import verbose, run_unittest, findfile +from test import support import unittest import xml.dom @@ -14,7 +14,7 @@ from xml.dom.minidom import parse, Node, Document, parseString from xml.dom.minidom import getDOMImplementation -tstfile = findfile("test.xml", subdir="xmltestdata") +tstfile = support.findfile("test.xml", subdir="xmltestdata") sample = ("<?xml version='1.0' encoding='us-ascii'?>\n" "<!DOCTYPE doc PUBLIC 'http://xml.python.org/public'" " 'http://xml.python.org/system' [\n" @@ -711,6 +711,57 @@ class MinidomTest(unittest.TestCase): def testClonePIDeep(self): self.check_clone_pi(1, "testClonePIDeep") + def check_clone_node_entity(self, clone_document): + # bpo-35052: Test user data handler in cloneNode() on a document with + # an entity + document = xml.dom.minidom.parseString(""" + <?xml version="1.0" ?> + <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" + "http://www.w3.org/TR/html4/strict.dtd" + [ <!ENTITY smile ":-)"> ] + > + <doc>Don't let entities make you frown ⌣</doc> + """.strip()) + + class Handler: + def handle(self, operation, key, data, src, dst): + self.operation = operation + self.key = key + self.data = data + self.src = src + self.dst = dst + + handler = Handler() + doctype = document.doctype + entity = doctype.entities['smile'] + entity.setUserData("key", "data", handler) + + if clone_document: + # clone Document + clone = document.cloneNode(deep=True) + + self.assertEqual(clone.documentElement.firstChild.wholeText, + "Don't let entities make you frown :-)") + operation = xml.dom.UserDataHandler.NODE_IMPORTED + dst = clone.doctype.entities['smile'] + else: + # clone DocumentType + with support.swap_attr(doctype, 'ownerDocument', None): + clone = doctype.cloneNode(deep=True) + + operation = xml.dom.UserDataHandler.NODE_CLONED + dst = clone.entities['smile'] + + self.assertEqual(handler.operation, operation) + self.assertEqual(handler.key, "key") + self.assertEqual(handler.data, "data") + self.assertIs(handler.src, entity) + self.assertIs(handler.dst, dst) + + def testCloneNodeEntity(self): + self.check_clone_node_entity(False) + self.check_clone_node_entity(True) + def testNormalize(self): doc = parseString("<doc/>") root = doc.documentElement @@ -1446,7 +1497,7 @@ class MinidomTest(unittest.TestCase): def test_main(): - run_unittest(MinidomTest) + support.run_unittest(MinidomTest) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 0350b3d..fc6d6e1 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -666,6 +666,13 @@ class MmapTests(unittest.TestCase): self.assertRaises(ValueError, m.write_byte, 'b') self.assertRaises(ValueError, m.write, 'abc') + def test_concat_repeat_exception(self): + m = mmap.mmap(-1, 16) + with self.assertRaises(TypeError): + m + m + with self.assertRaises(TypeError): + m * 2 + class LargeMmapTests(unittest.TestCase): diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py index 1071d7f..ff299fe 100644 --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -16,16 +16,16 @@ import logging import errno import weakref import test.script_helper -from test import test_support +from test import support from StringIO import StringIO -_multiprocessing = test_support.import_module('_multiprocessing') +_multiprocessing = support.import_module('_multiprocessing') # import threading after _multiprocessing to raise a more relevant error # message: "No module named _multiprocessing". _multiprocessing is not compiled # without thread support. import threading # Work around broken sem_open implementations -test_support.import_module('multiprocessing.synchronize') +support.import_module('multiprocessing.synchronize') import multiprocessing.dummy import multiprocessing.connection @@ -341,8 +341,8 @@ class _TestProcess(BaseTestCase): if self.TYPE == 'threads': self.skipTest('test not appropriate for {}'.format(self.TYPE)) - testfn = test_support.TESTFN - self.addCleanup(test_support.unlink, testfn) + testfn = support.TESTFN + self.addCleanup(support.unlink, testfn) for reason, code in (([1, 2, 3], 1), ('ignore this', 1)): p = self.Process(target=self._test_sys_exit, args=(reason, testfn)) @@ -640,7 +640,7 @@ class _TestQueue(BaseTestCase): p.join() def test_no_import_lock_contention(self): - with test_support.temp_cwd(): + with support.temp_cwd(): module_name = 'imported_by_an_imported_module' with open(module_name + '.py', 'w') as f: f.write("""if 1: @@ -652,7 +652,7 @@ class _TestQueue(BaseTestCase): q.close() """) - with test_support.DirsOnSysPath(os.getcwd()): + with support.DirsOnSysPath(os.getcwd()): try: __import__(module_name) except Queue.Empty: @@ -1135,6 +1135,16 @@ class _TestContainers(BaseTestCase): a.append('hello') self.assertEqual(f[:], [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 'hello']]) + def test_list_iter(self): + a = self.list(range(10)) + it = iter(a) + self.assertEqual(list(it), range(10)) + self.assertEqual(list(it), []) # exhausted + # list modified during iteration + it = iter(a) + a[0] = 100 + self.assertEqual(next(it), 100) + def test_dict(self): d = self.dict() indices = range(65, 70) @@ -1145,6 +1155,19 @@ class _TestContainers(BaseTestCase): self.assertEqual(sorted(d.values()), [chr(i) for i in indices]) self.assertEqual(sorted(d.items()), [(i, chr(i)) for i in indices]) + def test_dict_iter(self): + d = self.dict() + indices = range(65, 70) + for i in indices: + d[i] = chr(i) + it = iter(d) + self.assertEqual(list(it), indices) + self.assertEqual(list(it), []) # exhausted + # dictionary changed size during iteration + it = iter(d) + d.clear() + self.assertRaises(RuntimeError, next, it) + def test_namespace(self): n = self.Namespace() n.name = 'Bob' @@ -1260,10 +1283,10 @@ class _TestPool(BaseTestCase): self.assertRaises(SayWhenError, it.next) def test_imap_unordered(self): - it = self.pool.imap_unordered(sqr, range(1000)) - self.assertEqual(sorted(it), map(sqr, range(1000))) + it = self.pool.imap_unordered(sqr, range(100)) + self.assertEqual(sorted(it), map(sqr, range(100))) - it = self.pool.imap_unordered(sqr, range(1000), chunksize=53) + it = self.pool.imap_unordered(sqr, range(1000), chunksize=100) self.assertEqual(sorted(it), map(sqr, range(1000))) def test_imap_unordered_handle_iterable_exception(self): @@ -1517,10 +1540,10 @@ class _TestRemoteManager(BaseTestCase): #'hall\xc3\xa5 v\xc3\xa4rlden'] # UTF-8 ] result = values[:] - if test_support.have_unicode: + if support.have_unicode: #result[-1] = u'hall\xe5 v\xe4rlden' - uvalue = test_support.u(r'\u043f\u0440\u0438\u0432\u0456\u0442 ' - r'\u0441\u0432\u0456\u0442') + uvalue = support.u(r'\u043f\u0440\u0438\u0432\u0456\u0442 ' + r'\u0441\u0432\u0456\u0442') values.append(uvalue) result.append(uvalue) @@ -1538,7 +1561,7 @@ class _TestRemoteManager(BaseTestCase): authkey = os.urandom(32) manager = QueueManager( - address=(test.test_support.HOST, 0), authkey=authkey, serializer=SERIALIZER + address=(test.support.HOST, 0), authkey=authkey, serializer=SERIALIZER ) manager.start() @@ -1575,7 +1598,7 @@ class _TestManagerRestart(BaseTestCase): def test_rapid_restart(self): authkey = os.urandom(32) manager = QueueManager( - address=(test.test_support.HOST, 0), authkey=authkey, serializer=SERIALIZER) + address=(test.support.HOST, 0), authkey=authkey, serializer=SERIALIZER) srvr = manager.get_server() addr = srvr.address # Close the connection.Listener socket which gets opened as a part @@ -1786,13 +1809,13 @@ class _TestConnection(BaseTestCase): p = self.Process(target=self._writefd, args=(child_conn, b"foo")) p.daemon = True p.start() - with open(test_support.TESTFN, "wb") as f: + with open(support.TESTFN, "wb") as f: fd = f.fileno() if msvcrt: fd = msvcrt.get_osfhandle(fd) reduction.send_handle(conn, fd, p.pid) p.join() - with open(test_support.TESTFN, "rb") as f: + with open(support.TESTFN, "rb") as f: self.assertEqual(f.read(), b"foo") @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction") @@ -1811,7 +1834,7 @@ class _TestConnection(BaseTestCase): p = self.Process(target=self._writefd, args=(child_conn, b"bar", True)) p.daemon = True p.start() - with open(test_support.TESTFN, "wb") as f: + with open(support.TESTFN, "wb") as f: fd = f.fileno() for newfd in range(256, MAXFD): if not self._is_fd_assigned(newfd): @@ -1824,7 +1847,7 @@ class _TestConnection(BaseTestCase): finally: os.close(newfd) p.join() - with open(test_support.TESTFN, "rb") as f: + with open(support.TESTFN, "rb") as f: self.assertEqual(f.read(), b"bar") @classmethod @@ -2207,7 +2230,7 @@ class _TestFinalize(BaseTestCase): gc.set_threshold(5, 5, 5) threads = [threading.Thread(target=run_finalizers), threading.Thread(target=make_finalizers)] - with test_support.start_threads(threads): + with support.start_threads(threads): time.sleep(4.0) # Wait a bit to trigger race condition finish = True if exc: @@ -2635,7 +2658,7 @@ class TestFlags(unittest.TestCase): flags = (tuple(sys.flags), grandchild_flags) print(json.dumps(flags)) - @test_support.requires_unicode # XXX json needs unicode support + @support.requires_unicode # XXX json needs unicode support def test_flags(self): import json, subprocess # start child process using unusual flags @@ -2681,6 +2704,9 @@ class TestForkAwareThreadLock(unittest.TestCase): class TestIgnoreEINTR(unittest.TestCase): + # Sending CONN_MAX_SIZE bytes into a multiprocessing pipe must block + CONN_MAX_SIZE = max(support.PIPE_MAX_SIZE, support.SOCK_MAX_SIZE) + @classmethod def _test_ignore(cls, conn): def handler(signum, frame): @@ -2689,7 +2715,7 @@ class TestIgnoreEINTR(unittest.TestCase): conn.send('ready') x = conn.recv() conn.send(x) - conn.send_bytes(b'x'*(1024*1024)) # sending 1 MB should block + conn.send_bytes(b'x' * cls.CONN_MAX_SIZE) @unittest.skipUnless(hasattr(signal, 'SIGUSR1'), 'requires SIGUSR1') def test_ignore(self): @@ -2708,7 +2734,7 @@ class TestIgnoreEINTR(unittest.TestCase): self.assertEqual(conn.recv(), 1234) time.sleep(0.1) os.kill(p.pid, signal.SIGUSR1) - self.assertEqual(conn.recv_bytes(), b'x'*(1024*1024)) + self.assertEqual(conn.recv_bytes(), b'x' * self.CONN_MAX_SIZE) time.sleep(0.1) p.join() finally: @@ -2765,7 +2791,7 @@ def test_main(run=None): check_enough_semaphores() if run is None: - from test.test_support import run_unittest as run + from test.support import run_unittest as run util.get_temp_dir() # creates temp directory for use by all processes @@ -2790,7 +2816,7 @@ def test_main(run=None): # module during these tests is at least platform dependent and possibly # non-deterministic on any given platform. So we don't mind if the listed # warnings aren't actually raised. - with test_support.check_py3k_warnings( + with support.check_py3k_warnings( (".+__(get|set)slice__ has been removed", DeprecationWarning), (r"sys.exc_clear\(\) not supported", DeprecationWarning), quiet=True): diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py index 34b9cbb..73974a9 100644 --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -263,21 +263,19 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase): # An absolutely minimal test of position information. Better # tests would be a big project. code = "def f(x):\n return x + 1" - st1 = parser.suite(code) - st2 = st1.totuple(line_info=1, col_info=1) + st = parser.suite(code) def walk(tree): node_type = tree[0] next = tree[1] - if isinstance(next, tuple): + if isinstance(next, (tuple, list)): for elt in tree[1:]: for x in walk(elt): yield x else: yield tree - terminals = list(walk(st2)) - self.assertEqual([ + expected = [ (1, 'def', 1, 0), (1, 'f', 1, 4), (7, '(', 1, 5), @@ -293,8 +291,25 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase): (4, '', 2, 16), (6, '', 2, -1), (4, '', 2, -1), - (0, '', 2, -1)], - terminals) + (0, '', 2, -1), + ] + + self.assertEqual(list(walk(st.totuple(line_info=True, col_info=True))), + expected) + self.assertEqual(list(walk(st.totuple())), + [(t, n) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.totuple(line_info=True))), + [(t, n, l) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.totuple(col_info=True))), + [(t, n, c) for t, n, l, c in expected]) + self.assertEqual(list(walk(st.tolist(line_info=True, col_info=True))), + [list(x) for x in expected]) + self.assertEqual(list(walk(parser.st2tuple(st, line_info=True, + col_info=True))), + expected) + self.assertEqual(list(walk(parser.st2list(st, line_info=True, + col_info=True))), + [list(x) for x in expected]) # diff --git a/Lib/test/test_pkg.py b/Lib/test/test_pkg.py index 5f1659b..ff05dce 100644 --- a/Lib/test/test_pkg.py +++ b/Lib/test/test_pkg.py @@ -134,7 +134,7 @@ class Test(unittest.TestCase): s = """ from t2 import * - self.assertTrue(dir(), ['self', 'sub']) + self.assertEqual(dir(), ['self', 'sub']) """ self.run_code(s) diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index f754550..542763d 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -4,7 +4,7 @@ import unittest import platform import subprocess -from test import test_support +from test import support class PlatformTest(unittest.TestCase): def test_architecture(self): @@ -18,7 +18,7 @@ class PlatformTest(unittest.TestCase): p = subprocess.Popen(cmd, stdout=subprocess.PIPE) return p.communicate() real = os.path.realpath(sys.executable) - link = os.path.abspath(test_support.TESTFN) + link = os.path.abspath(support.TESTFN) os.symlink(real, link) try: self.assertEqual(get(real), get(link)) @@ -163,7 +163,7 @@ class PlatformTest(unittest.TestCase): # using it, per # http://blogs.msdn.com/david.wang/archive/2006/03/26/HOWTO-Detect-Process-Bitness.aspx try: - with test_support.EnvironmentVarGuard() as environ: + with support.EnvironmentVarGuard() as environ: if 'PROCESSOR_ARCHITEW6432' in environ: del environ['PROCESSOR_ARCHITEW6432'] environ['PROCESSOR_ARCHITECTURE'] = 'foo' @@ -247,7 +247,6 @@ class PlatformTest(unittest.TestCase): res = platform.dist() def test_libc_ver(self): - import os if os.path.isdir(sys.executable) and \ os.path.exists(sys.executable+'.exe'): # Cygwin horror @@ -256,6 +255,13 @@ class PlatformTest(unittest.TestCase): executable = sys.executable res = platform.libc_ver(executable) + self.addCleanup(support.unlink, support.TESTFN) + with open(support.TESTFN, 'wb') as f: + f.write(b'x'*(16384-10)) + f.write(b'GLIBC_1.23.4\0GLIBC_1.9\0GLIBC_1.21\0') + self.assertEqual(platform.libc_ver(support.TESTFN), + ('glibc', '1.23.4')) + def test_parse_release_file(self): for input, output in ( @@ -275,7 +281,7 @@ class PlatformTest(unittest.TestCase): def test_main(): - test_support.run_unittest( + support.run_unittest( PlatformTest ) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index bce4e21..c4283b6 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -199,6 +199,7 @@ class PosixTester(unittest.TestCase): def test_fdopen_directory(self): try: fd = os.open('.', os.O_RDONLY) + self.addCleanup(os.close, fd) except OSError as e: self.assertEqual(e.errno, errno.EACCES) self.skipTest("system cannot open directories") diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index deaa577..18ea2e4 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -262,34 +262,56 @@ class PosixPathTest(unittest.TestCase): def test_expanduser(self): self.assertEqual(posixpath.expanduser("foo"), "foo") - with test_support.EnvironmentVarGuard() as env: + + def test_expanduser_home_envvar(self): + with support.EnvironmentVarGuard() as env: + env['HOME'] = '/home/victor' + self.assertEqual(posixpath.expanduser("~"), "/home/victor") + + # expanduser() strips trailing slash + env['HOME'] = '/home/victor/' + self.assertEqual(posixpath.expanduser("~"), "/home/victor") + for home in '/', '', '//', '///': env['HOME'] = home self.assertEqual(posixpath.expanduser("~"), "/") self.assertEqual(posixpath.expanduser("~/"), "/") self.assertEqual(posixpath.expanduser("~/foo"), "/foo") - try: - import pwd - except ImportError: - pass - else: - self.assertIsInstance(posixpath.expanduser("~/"), basestring) - # if home directory == root directory, this test makes no sense - if posixpath.expanduser("~") != '/': - self.assertEqual( - posixpath.expanduser("~") + "/", - posixpath.expanduser("~/") - ) - self.assertIsInstance(posixpath.expanduser("~root/"), basestring) - self.assertIsInstance(posixpath.expanduser("~foo/"), basestring) - with test_support.EnvironmentVarGuard() as env: - # expanduser should fall back to using the password database - del env['HOME'] - home = pwd.getpwuid(os.getuid()).pw_dir - # $HOME can end with a trailing /, so strip it (see #17809) - home = home.rstrip("/") or '/' - self.assertEqual(posixpath.expanduser("~"), home) + def test_expanduser_pwd(self): + pwd = support.import_module('pwd') + + self.assertIsInstance(posixpath.expanduser("~/"), str) + + # if home directory == root directory, this test makes no sense + if posixpath.expanduser("~") != '/': + self.assertEqual( + posixpath.expanduser("~") + "/", + posixpath.expanduser("~/") + ) + self.assertIsInstance(posixpath.expanduser("~root/"), str) + self.assertIsInstance(posixpath.expanduser("~foo/"), str) + + with support.EnvironmentVarGuard() as env: + # expanduser should fall back to using the password database + del env['HOME'] + + home = pwd.getpwuid(os.getuid()).pw_dir + # $HOME can end with a trailing /, so strip it (see #17809) + home = home.rstrip("/") or '/' + self.assertEqual(posixpath.expanduser("~"), home) + + # bpo-10496: If the HOME environment variable is not set and the + # user (current identifier or name in the path) doesn't exist in + # the password database (pwd.getuid() or pwd.getpwnam() fail), + # expanduser() must return the path unchanged. + def raise_keyerror(*args): + raise KeyError + + with support.swap_attr(pwd, 'getpwuid', raise_keyerror), \ + support.swap_attr(pwd, 'getpwnam', raise_keyerror): + for path in ('~', '~/.local', '~vstinner/'): + self.assertEqual(posixpath.expanduser(path), path) def test_normpath(self): self.assertEqual(posixpath.normpath(""), ".") @@ -474,12 +496,10 @@ class PosixPathTest(unittest.TestCase): finally: os.getcwd = real_getcwd - @test_support.requires_unicode + @unittest.skipUnless(test_support.FS_NONASCII, 'need test_support.FS_NONASCII') def test_expandvars_nonascii_word(self): encoding = sys.getfilesystemencoding() - # Non-ASCII word characters - letters = test_support.u(r'\xe6\u0130\u0141\u03c6\u041a\u05d0\u062a\u0e01') - uwnonascii = letters.encode(encoding, 'ignore').decode(encoding)[:3] + uwnonascii = test_support.FS_NONASCII swnonascii = uwnonascii.encode(encoding) if not swnonascii: self.skipTest('Needs non-ASCII word characters') diff --git a/Lib/test/test_pty.py b/Lib/test/test_pty.py index f623aa0..0eb31fd 100644 --- a/Lib/test/test_pty.py +++ b/Lib/test/test_pty.py @@ -70,14 +70,11 @@ class PtyTest(unittest.TestCase): def setUp(self): # isatty() and close() can hang on some platforms. Set an alarm # before running the test to make sure we don't hang forever. - self.old_alarm = signal.signal(signal.SIGALRM, self.handle_sig) + old_alarm = signal.signal(signal.SIGALRM, self.handle_sig) + self.addCleanup(signal.signal, signal.SIGALRM, old_alarm) + self.addCleanup(signal.alarm, 0) signal.alarm(10) - def tearDown(self): - # remove alarm, restore old alarm handler - signal.alarm(0) - signal.signal(signal.SIGALRM, self.old_alarm) - def handle_sig(self, sig, frame): self.fail("isatty hung") diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 988a72c..872ba64 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -17,6 +17,8 @@ import tempfile import textwrap import unittest from test import support +# Use utils alias to use the same code for TestUtils in master and 2.7 branches +import regrtest as utils Py_DEBUG = hasattr(sys, 'getobjects') @@ -49,11 +51,24 @@ class BaseTestCase(unittest.TestCase): self.tmptestdir = tempfile.mkdtemp() self.addCleanup(support.rmtree, self.tmptestdir) - def create_test(self, name=None, code=''): + def create_test(self, name=None, code=None): if not name: name = 'noop%s' % BaseTestCase.TEST_UNIQUE_ID BaseTestCase.TEST_UNIQUE_ID += 1 + if code is None: + code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_empty_test(self): + pass + + def test_main(): + support.run_unittest(Tests) + """) + # test_regrtest cannot be run twice in parallel because # of setUp() and create_test() name = self.TESTNAME_PREFIX + name @@ -92,6 +107,7 @@ class BaseTestCase(unittest.TestCase): def check_executed_tests(self, output, tests, skipped=(), failed=(), env_changed=(), omitted=(), + rerun=(), no_test_ran=(), randomize=False, interrupted=False, fail_env_changed=False): if isinstance(tests, str): @@ -104,6 +120,10 @@ class BaseTestCase(unittest.TestCase): env_changed = [env_changed] if isinstance(omitted, str): omitted = [omitted] + if isinstance(rerun, str): + rerun = [rerun] + if isinstance(no_test_ran, str): + no_test_ran = [no_test_ran] executed = self.parse_executed_tests(output) if randomize: @@ -138,8 +158,16 @@ class BaseTestCase(unittest.TestCase): regex = list_regex('%s test%s omitted', omitted) self.check_line(output, regex) + if rerun: + regex = list_regex('%s re-run test%s', rerun) + self.check_line(output, regex) + self.check_line(output, "Re-running failed tests in verbose mode") + for name in rerun: + regex = "Re-running test %r in verbose mode" % name + self.check_line(output, regex) + good = (len(tests) - len(skipped) - len(failed) - - len(omitted) - len(env_changed)) + - len(omitted) - len(env_changed) - len(no_test_ran)) if good: regex = r'%s test%s OK\.$' % (good, plural(good)) if not skipped and not failed and good > 1: @@ -149,14 +177,23 @@ class BaseTestCase(unittest.TestCase): if interrupted: self.check_line(output, 'Test suite interrupted by signal SIGINT.') + result = [] if failed: - result = 'FAILURE' - elif interrupted: - result = 'INTERRUPTED' + result.append('FAILURE') elif fail_env_changed and env_changed: - result = 'ENV CHANGED' - else: - result = 'SUCCESS' + result.append('ENV CHANGED') + if interrupted: + result.append('INTERRUPTED') + if not any((good, result, failed, interrupted, skipped, + env_changed, fail_env_changed)): + result.append("NO TEST RUN") + elif not result: + result.append('SUCCESS') + result = ', '.join(result) + if rerun: + self.check_line(output, 'Tests result: %s' % result) + result = 'FAILURE then %s' % result + self.check_line(output, 'Tests result: %s' % result) def parse_random_seed(self, output): @@ -340,7 +377,17 @@ class ArgsTestCase(BaseTestCase): # test -u command line option tests = {} for resource in ('audio', 'network'): - code = 'from test import support\nsupport.requires(%r)' % resource + code = textwrap.dedent(""" + from test import support; support.requires(%r) + import unittest + class PassingTest(unittest.TestCase): + def test_pass(self): + pass + + def test_main(): + support.run_unittest(PassingTest) + """ % resource) + tests[resource] = self.create_test(resource, code) test_names = sorted(tests.values()) @@ -511,6 +558,24 @@ class ArgsTestCase(BaseTestCase): """) self.check_leak(code, 'references') + @unittest.skipUnless(Py_DEBUG, 'need a debug build') + def test_huntrleaks_fd_leak(self): + # test --huntrleaks for file descriptor leak + code = textwrap.dedent(""" + import os + import unittest + from test import support + + class FDLeakTest(unittest.TestCase): + def test_leak(self): + fd = os.open(__file__, os.O_RDONLY) + # bug: never close the file descriptor + + def test_main(): + support.run_unittest(FDLeakTest) + """) + self.check_leak(code, 'file descriptors') + def test_list_tests(self): # test --list-tests tests = [self.create_test() for i in range(5)] @@ -607,6 +672,9 @@ class ArgsTestCase(BaseTestCase): subset = ['test_method1', 'test_method3'] self.assertEqual(methods, subset) + # bpo-34021: The test fails randomly for an unknown reason + # on "x86 Windows XP VS9.0 2.7" buildbot worker. + @unittest.skipIf(sys.platform == "win32", "test fails randomly on Windows") def test_env_changed(self): code = textwrap.dedent(""" import unittest @@ -630,9 +698,135 @@ class ArgsTestCase(BaseTestCase): self.check_executed_tests(output, [testname], env_changed=testname, fail_env_changed=True) + def test_rerun_fail(self): + code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_bug(self): + # test always fail + self.fail("bug") + + def test_main(): + support.run_unittest(Tests) + """) + testname = self.create_test(code=code) + + output = self.run_tests("-w", testname, exitcode=2) + self.check_executed_tests(output, [testname], + failed=testname, rerun=testname) + + def test_no_tests_ran(self): + code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_bug(self): + pass + + def test_main(): + support.run_unittest(Tests) + """) + testname = self.create_test(code=code) + + output = self.run_tests("-m", "nosuchtest", testname, exitcode=0) + self.check_executed_tests(output, [testname], no_test_ran=testname) + + def test_no_tests_ran_skip(self): + code = textwrap.dedent(""" + import unittest + + class Tests(unittest.TestCase): + def test_skipped(self): + self.skipTest("because") + """) + testname = self.create_test(code=code) + + output = self.run_tests(testname, exitcode=0) + self.check_executed_tests(output, [testname]) + + def test_no_tests_ran_multiple_tests_nonexistent(self): + code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_bug(self): + pass + + def test_main(): + support.run_unittest(Tests) + """) + testname = self.create_test(code=code) + testname2 = self.create_test(code=code) + + output = self.run_tests("-m", "nosuchtest", + testname, testname2, + exitcode=0) + self.check_executed_tests(output, [testname, testname2], + no_test_ran=[testname, testname2]) + + def test_no_test_ran_some_test_exist_some_not(self): + code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_bug(self): + pass + + def test_main(): + support.run_unittest(Tests) + """) + testname = self.create_test(code=code) + other_code = textwrap.dedent(""" + import unittest + from test import support + + class Tests(unittest.TestCase): + def test_other_bug(self): + pass + + def test_main(): + support.run_unittest(Tests) + """) + testname2 = self.create_test(code=other_code) + + output = self.run_tests("-m", "nosuchtest", "-m", "test_other_bug", + testname, testname2, + exitcode=0) + self.check_executed_tests(output, [testname, testname2], + no_test_ran=[testname]) + + +class TestUtils(unittest.TestCase): + def test_format_duration(self): + self.assertEqual(utils.format_duration(0), + '0 ms') + self.assertEqual(utils.format_duration(1e-9), + '1 ms') + self.assertEqual(utils.format_duration(10e-3), + '10 ms') + self.assertEqual(utils.format_duration(1.5), + '1 sec 500 ms') + self.assertEqual(utils.format_duration(1), + '1 sec') + self.assertEqual(utils.format_duration(2 * 60), + '2 min') + self.assertEqual(utils.format_duration(2 * 60 + 1), + '2 min 1 sec') + self.assertEqual(utils.format_duration(3 * 3600), + '3 hour') + self.assertEqual(utils.format_duration(3 * 3600 + 2 * 60 + 1), + '3 hour 2 min') + self.assertEqual(utils.format_duration(3 * 3600 + 1), + '3 hour 1 sec') + def test_main(): - support.run_unittest(ProgramsTestCase, ArgsTestCase) + support.run_unittest(ProgramsTestCase, ArgsTestCase, TestUtils) if __name__ == "__main__": diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py index 8ed5d89..ba7ccf8 100644 --- a/Lib/test/test_robotparser.py +++ b/Lib/test/test_robotparser.py @@ -136,6 +136,31 @@ Disallow: /cyberworld/map/ bad = ['/cyberworld/map/index.html'] +class StringFormattingTest(BaseRobotTest, unittest.TestCase): + robots_txt = """\ +User-agent: * +Crawl-delay: 1 +Request-rate: 3/15 +Disallow: /cyberworld/map/ # This is an infinite virtual URL space + +# Cybermapper knows where to go. +User-agent: cybermapper +Disallow: /some/path + """ + + expected_output = """\ +User-agent: cybermapper +Disallow: /some/path + +User-agent: * +Disallow: /cyberworld/map/ + +""" + + def test_string_formatting(self): + self.assertEqual(str(self.parser), self.expected_output) + + class RobotHandler(BaseHTTPRequestHandler): def do_GET(self): @@ -226,6 +251,7 @@ def test_main(): UseFirstUserAgentWildcardTest, EmptyQueryStringTest, DefaultEntryTest, + StringFormattingTest, PasswordProtectedSiteTestCase, NetworkTestCase) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py index 8338712..ca89fe5 100644 --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -543,6 +543,8 @@ class TestShutil(unittest.TestCase): subprocess.check_output(zip_cmd, stderr=subprocess.STDOUT) except subprocess.CalledProcessError as exc: details = exc.output + if 'unrecognized option: t' in details: + self.skipTest("unzip doesn't support -t") msg = "{}\n\n**Unzip Output**\n{}" self.fail(msg.format(exc, details)) diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py index 5cff13a..5f4f579 100644 --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -138,6 +138,8 @@ class InterProcessSignalTests(unittest.TestCase): else: self.fail("pause returned of its own accord, and the signal" " didn't arrive after another second.") + finally: + signal.alarm(0) # Issue 3864. Unknown if this affects earlier versions of freebsd also. @unittest.skipIf(sys.platform=='freebsd6', @@ -246,11 +248,15 @@ class WakeupSignalTests(unittest.TestCase): import select signal.alarm(1) - before_time = time.time() - # We attempt to get a signal during the sleep, - # before select is called - time.sleep(self.TIMEOUT_FULL) - mid_time = time.time() + try: + before_time = time.time() + # We attempt to get a signal during the sleep, + # before select is called + time.sleep(self.TIMEOUT_FULL) + mid_time = time.time() + finally: + signal.alarm(0) + self.assertTrue(mid_time - before_time < self.TIMEOUT_HALF) select.select([self.read], [], [], self.TIMEOUT_FULL) after_time = time.time() @@ -260,11 +266,15 @@ class WakeupSignalTests(unittest.TestCase): import select signal.alarm(1) - before_time = time.time() - # We attempt to get a signal during the select call - self.assertRaises(select.error, select.select, - [self.read], [], [], self.TIMEOUT_FULL) - after_time = time.time() + try: + before_time = time.time() + # We attempt to get a signal during the select call + self.assertRaises(select.error, select.select, + [self.read], [], [], self.TIMEOUT_FULL) + after_time = time.time() + finally: + signal.alarm(0) + self.assertTrue(after_time - before_time < self.TIMEOUT_HALF) def setUp(self): diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index 9569135..b4384ee 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -7,6 +7,7 @@ executing have not been removed. import unittest from test.test_support import run_unittest, TESTFN, EnvironmentVarGuard from test.test_support import captured_output +from test import support import __builtin__ import errno import os @@ -241,6 +242,7 @@ class HelperFunctionsTests(unittest.TestCase): # the call sets USER_BASE *and* USER_SITE self.assertEqual(site.USER_SITE, user_site) self.assertTrue(user_site.startswith(site.USER_BASE), user_site) + self.assertEqual(site.USER_BASE, site.getuserbase()) def test_getsitepackages(self): site.PREFIXES = ['xoxo'] @@ -265,6 +267,48 @@ class HelperFunctionsTests(unittest.TestCase): wanted = os.path.join('xoxo', 'lib', 'site-packages') self.assertEqual(dirs[1], wanted) + def test_no_home_directory(self): + # bpo-10496: getuserbase() and getusersitepackages() must not fail if + # the current user has no home directory (if expanduser() returns the + # path unchanged). + site.USER_SITE = None + site.USER_BASE = None + sysconfig._CONFIG_VARS = None + + with EnvironmentVarGuard() as environ, \ + support.swap_attr(os.path, 'expanduser', lambda path: path): + + del environ['PYTHONUSERBASE'] + del environ['APPDATA'] + + user_base = site.getuserbase() + self.assertTrue(user_base.startswith('~' + os.sep), + user_base) + + user_site = site.getusersitepackages() + self.assertTrue(user_site.startswith(user_base), user_site) + + def fake_isdir(path): + fake_isdir.arg = path + return False + fake_isdir.arg = None + + def must_not_be_called(*args): + raise AssertionError + + with support.swap_attr(os.path, 'isdir', fake_isdir), \ + support.swap_attr(site, 'addsitedir', must_not_be_called), \ + support.swap_attr(site, 'ENABLE_USER_SITE', True): + + # addusersitepackages() must not add user_site to sys.path + # if it is not an existing directory + known_paths = set() + site.addusersitepackages(known_paths) + + self.assertEqual(fake_isdir.arg, user_site) + self.assertFalse(known_paths) + + class PthFile(object): """Helper class for handling testing of .pth files""" diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index 1bb6690..703b631 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -306,12 +306,14 @@ class TooLongLineTests(unittest.TestCase): self.sock.settimeout(15) self.port = test_support.bind_port(self.sock) servargs = (self.evt, self.respdata, self.sock) - threading.Thread(target=server, args=servargs).start() + self.thread = threading.Thread(target=server, args=servargs) + self.thread.start() self.evt.wait() self.evt.clear() def tearDown(self): self.evt.wait() + self.thread.join() sys.stdout = self.old_stdout def testLineTooLong(self): diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index ca53899..988e12a 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -21,6 +21,9 @@ except ImportError: _socket = None +MAIN_TIMEOUT = 60.0 + + def try_address(host, port=0, family=socket.AF_INET): """Try to bind a socket on the given host:port and return True if that has been possible.""" @@ -947,6 +950,7 @@ class BasicSocketPairTest(SocketPairTest): class NonBlockingTCPTests(ThreadedTCPSocketTest): def __init__(self, methodName='runTest'): + self.event = threading.Event() ThreadedTCPSocketTest.__init__(self, methodName=methodName) def testSetBlocking(self): @@ -982,21 +986,27 @@ class NonBlockingTCPTests(ThreadedTCPSocketTest): def testAccept(self): # Testing non-blocking accept self.serv.setblocking(0) - try: - conn, addr = self.serv.accept() - except socket.error: - pass - else: - self.fail("Error trying to do non-blocking accept.") - read, write, err = select.select([self.serv], [], []) - if self.serv in read: + + # connect() didn't start: non-blocking accept() fails + with self.assertRaises(socket.error): conn, addr = self.serv.accept() - conn.close() - else: + + self.event.set() + + read, write, err = select.select([self.serv], [], [], MAIN_TIMEOUT) + if self.serv not in read: self.fail("Error trying to do accept after select.") + # connect() completed: non-blocking accept() doesn't block + conn, addr = self.serv.accept() + self.addCleanup(conn.close) + self.assertIsNone(conn.gettimeout()) + def _testAccept(self): - time.sleep(0.1) + # don't connect before event is set to check + # that non-blocking accept() raises socket.error + self.event.wait() + self.cli.connect((HOST, self.port)) def testConnect(self): @@ -1011,25 +1021,32 @@ class NonBlockingTCPTests(ThreadedTCPSocketTest): def testRecv(self): # Testing non-blocking recv conn, addr = self.serv.accept() + self.addCleanup(conn.close) conn.setblocking(0) - try: - msg = conn.recv(len(MSG)) - except socket.error: - pass - else: - self.fail("Error trying to do non-blocking recv.") - read, write, err = select.select([conn], [], []) - if conn in read: + + # the server didn't send data yet: non-blocking recv() fails + with self.assertRaises(socket.error): msg = conn.recv(len(MSG)) - conn.close() - self.assertEqual(msg, MSG) - else: + + self.event.set() + + read, write, err = select.select([conn], [], [], MAIN_TIMEOUT) + if conn not in read: self.fail("Error during select call to non-blocking socket.") + # the server sent data yet: non-blocking recv() doesn't block + msg = conn.recv(len(MSG)) + self.assertEqual(msg, MSG) + def _testRecv(self): self.cli.connect((HOST, self.port)) - time.sleep(0.1) - self.cli.send(MSG) + + # don't send anything before event is set to check + # that non-blocking recv() raises socket.error + self.event.wait() + + # send data: recv() will no longer block + self.cli.sendall(MSG) @unittest.skipUnless(thread, 'Threading required for this test.') class FileObjectClassTestCase(SocketConnectedTest): @@ -1534,8 +1551,8 @@ class TCPTimeoutTest(SocketTCPTest): raise Alarm old_alarm = signal.signal(signal.SIGALRM, alarm_handler) try: - signal.alarm(2) # POSIX allows alarm to be up to 1 second early try: + signal.alarm(2) # POSIX allows alarm to be up to 1 second early foo = self.serv.accept() except socket.timeout: self.fail("caught timeout instead of Alarm") diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py index 0087b9e..847859d 100644 --- a/Lib/test/test_socketserver.py +++ b/Lib/test/test_socketserver.py @@ -88,12 +88,13 @@ class SocketServerTest(unittest.TestCase): """Test all socket servers.""" def setUp(self): + self.addCleanup(signal_alarm, 0) signal_alarm(60) # Kill deadlocks after 60 seconds. self.port_seed = 0 self.test_files = [] def tearDown(self): - signal_alarm(0) # Didn't deadlock. + self.doCleanups() reap_children() for fn in self.test_files: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index b59fe73..0eecc78 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -72,10 +72,18 @@ NONEXISTINGCERT = data_file("XXXnonexisting.pem") BADKEY = data_file("badkey.pem") NOKIACERT = data_file("nokia.pem") NULLBYTECERT = data_file("nullbytecert.pem") +TALOS_INVALID_CRLDP = data_file("talos-2019-0758.pem") -DHFILE = data_file("dh1024.pem") +DHFILE = data_file("ffdh3072.pem") BYTES_DHFILE = DHFILE.encode(sys.getfilesystemencoding()) +# Not defined in all versions of OpenSSL +OP_NO_COMPRESSION = getattr(ssl, "OP_NO_COMPRESSION", 0) +OP_SINGLE_DH_USE = getattr(ssl, "OP_SINGLE_DH_USE", 0) +OP_SINGLE_ECDH_USE = getattr(ssl, "OP_SINGLE_ECDH_USE", 0) +OP_CIPHER_SERVER_PREFERENCE = getattr(ssl, "OP_CIPHER_SERVER_PREFERENCE", 0) +OP_ENABLE_MIDDLEBOX_COMPAT = getattr(ssl, "OP_ENABLE_MIDDLEBOX_COMPAT", 0) + def handle_error(prefix): exc_format = ' '.join(traceback.format_exception(*sys.exc_info())) @@ -201,9 +209,9 @@ class BasicSocketTests(unittest.TestCase): (('commonName', 'localhost'),)) ) # Note the next three asserts will fail if the keys are regenerated - self.assertEqual(p['notAfter'], asn1time('Oct 5 23:01:56 2020 GMT')) - self.assertEqual(p['notBefore'], asn1time('Oct 8 23:01:56 2010 GMT')) - self.assertEqual(p['serialNumber'], 'D7C7381919AFC24E') + self.assertEqual(p['notAfter'], asn1time('Aug 26 14:23:15 2028 GMT')) + self.assertEqual(p['notBefore'], asn1time('Aug 29 14:23:15 2018 GMT')) + self.assertEqual(p['serialNumber'], '98A7CF88C74A32ED') self.assertEqual(p['subject'], ((('countryName', 'XY'),), (('localityName', 'Castle Anthrax'),), @@ -227,6 +235,27 @@ class BasicSocketTests(unittest.TestCase): self.assertEqual(p['crlDistributionPoints'], ('http://SVRIntl-G3-crl.verisign.com/SVRIntlG3.crl',)) + def test_parse_cert_CVE_2019_5010(self): + p = ssl._ssl._test_decode_cert(TALOS_INVALID_CRLDP) + if support.verbose: + sys.stdout.write("\n" + pprint.pformat(p) + "\n") + self.assertEqual( + p, + { + 'issuer': ( + (('countryName', 'UK'),), (('commonName', 'cody-ca'),)), + 'notAfter': 'Jun 14 18:00:58 2028 GMT', + 'notBefore': 'Jun 18 18:00:58 2018 GMT', + 'serialNumber': '02', + 'subject': ((('countryName', 'UK'),), + (('commonName', + 'codenomicon-vm-2.test.lal.cisco.com'),)), + 'subjectAltName': ( + ('DNS', 'codenomicon-vm-2.test.lal.cisco.com'),), + 'version': 3 + } + ) + def test_parse_cert_CVE_2013_4238(self): p = ssl._ssl._test_decode_cert(NULLBYTECERT) if support.verbose: @@ -341,6 +370,7 @@ class BasicSocketTests(unittest.TestCase): self.assertRaises(socket.error, ss.recvfrom_into, bytearray(b'x'), 1) self.assertRaises(socket.error, ss.send, b'x') self.assertRaises(socket.error, ss.sendto, b'x', ('0.0.0.0', 0)) + self.assertRaises(NotImplementedError, ss.dup) def test_timeout(self): # Issue #8524: when creating an SSL socket, the timeout of the @@ -775,8 +805,10 @@ class ContextTests(unittest.TestCase): ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) # OP_ALL | OP_NO_SSLv2 | OP_NO_SSLv3 is the default value default = (ssl.OP_ALL | ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3) - if not IS_LIBRESSL and ssl.OPENSSL_VERSION_INFO >= (1, 1, 0): - default |= ssl.OP_NO_COMPRESSION + # SSLContext also enables these by default + default |= (OP_NO_COMPRESSION | OP_CIPHER_SERVER_PREFERENCE | + OP_SINGLE_DH_USE | OP_SINGLE_ECDH_USE | + OP_ENABLE_MIDDLEBOX_COMPAT) self.assertEqual(default, ctx.options) ctx.options |= ssl.OP_NO_TLSv1 self.assertEqual(default | ssl.OP_NO_TLSv1, ctx.options) @@ -989,6 +1021,13 @@ class ContextTests(unittest.TestCase): def test_load_dh_params(self): + filename = u'dhpäräm.pem' + fs_encoding = sys.getfilesystemencoding() + try: + filename.encode(fs_encoding) + except UnicodeEncodeError: + self.skipTest("filename %r cannot be encoded to the filesystem encoding %r" % (filename, fs_encoding)) + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) ctx.load_dh_params(DHFILE) if os.name != 'nt': @@ -1001,7 +1040,7 @@ class ContextTests(unittest.TestCase): with self.assertRaises(ssl.SSLError) as cm: ctx.load_dh_params(CERTFILE) with support.temp_dir() as d: - fname = os.path.join(d, u'dhpäräm.pem') + fname = os.path.join(d, filename) shutil.copy(DHFILE, fname) ctx.load_dh_params(fname) @@ -1148,16 +1187,29 @@ class ContextTests(unittest.TestCase): stats["x509"] += 1 self.assertEqual(ctx.cert_store_stats(), stats) + def _assert_context_options(self, ctx): + self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + if OP_NO_COMPRESSION != 0: + self.assertEqual(ctx.options & OP_NO_COMPRESSION, + OP_NO_COMPRESSION) + if OP_SINGLE_DH_USE != 0: + self.assertEqual(ctx.options & OP_SINGLE_DH_USE, + OP_SINGLE_DH_USE) + if OP_SINGLE_ECDH_USE != 0: + self.assertEqual(ctx.options & OP_SINGLE_ECDH_USE, + OP_SINGLE_ECDH_USE) + if OP_CIPHER_SERVER_PREFERENCE != 0: + self.assertEqual(ctx.options & OP_CIPHER_SERVER_PREFERENCE, + OP_CIPHER_SERVER_PREFERENCE) + def test_create_default_context(self): ctx = ssl.create_default_context() + self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertTrue(ctx.check_hostname) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) - self.assertEqual( - ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), - getattr(ssl, "OP_NO_COMPRESSION", 0), - ) + self._assert_context_options(ctx) + with open(SIGNING_CA) as f: cadata = f.read().decode("ascii") @@ -1165,40 +1217,24 @@ class ContextTests(unittest.TestCase): cadata=cadata) self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) - self.assertEqual( - ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), - getattr(ssl, "OP_NO_COMPRESSION", 0), - ) + self._assert_context_options(ctx) ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) - self.assertEqual( - ctx.options & getattr(ssl, "OP_NO_COMPRESSION", 0), - getattr(ssl, "OP_NO_COMPRESSION", 0), - ) - self.assertEqual( - ctx.options & getattr(ssl, "OP_SINGLE_DH_USE", 0), - getattr(ssl, "OP_SINGLE_DH_USE", 0), - ) - self.assertEqual( - ctx.options & getattr(ssl, "OP_SINGLE_ECDH_USE", 0), - getattr(ssl, "OP_SINGLE_ECDH_USE", 0), - ) + self._assert_context_options(ctx) def test__create_stdlib_context(self): ctx = ssl._create_stdlib_context() self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) self.assertFalse(ctx.check_hostname) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self._assert_context_options(ctx) ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1) self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self._assert_context_options(ctx) ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1, cert_reqs=ssl.CERT_REQUIRED, @@ -1206,12 +1242,12 @@ class ContextTests(unittest.TestCase): self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) self.assertTrue(ctx.check_hostname) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self._assert_context_options(ctx) ctx = ssl._create_stdlib_context(purpose=ssl.Purpose.CLIENT_AUTH) self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) - self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + self._assert_context_options(ctx) def test__https_verify_certificates(self): # Unit test to check the contect factory mapping @@ -1663,23 +1699,43 @@ else: self.sock, server_side=True) self.server.selected_npn_protocols.append(self.sslconn.selected_npn_protocol()) self.server.selected_alpn_protocols.append(self.sslconn.selected_alpn_protocol()) - except socket.error as e: - # We treat ConnectionResetError as though it were an - # SSLError - OpenSSL on Ubuntu abruptly closes the - # connection when asked to use an unsupported protocol. - # - # XXX Various errors can have happened here, for example - # a mismatching protocol version, an invalid certificate, - # or a low-level bug. This should be made more discriminating. - if not isinstance(e, ssl.SSLError) and e.errno != errno.ECONNRESET: - raise - self.server.conn_errors.append(e) - if self.server.chatty: - handle_error("\n server: bad connection attempt from " + repr(self.addr) + ":\n") - self.running = False - self.server.stop() - self.close() - return False + except (ssl.SSLError, socket.error, OSError) as e: + if e.errno in (errno.ECONNRESET, errno.EPIPE, errno.ESHUTDOWN): + # Mimick Python 3: + # + # except (ConnectionResetError, BrokenPipeError): + # + # We treat ConnectionResetError as though it were an + # SSLError - OpenSSL on Ubuntu abruptly closes the + # connection when asked to use an unsupported protocol. + # + # BrokenPipeError is raised in TLS 1.3 mode, when OpenSSL + # tries to send session tickets after handshake. + # https://github.com/openssl/openssl/issues/6342 + self.server.conn_errors.append(str(e)) + if self.server.chatty: + handle_error( + "\n server: bad connection attempt from " + + repr(self.addr) + ":\n") + self.running = False + self.close() + return False + else: + # OSError may occur with wrong protocols, e.g. both + # sides use PROTOCOL_TLS_SERVER. + # + # XXX Various errors can have happened here, for example + # a mismatching protocol version, an invalid certificate, + # or a low-level bug. This should be made more discriminating. + if not isinstance(e, ssl.SSLError) and e.errno != errno.ECONNRESET: + raise + self.server.conn_errors.append(e) + if self.server.chatty: + handle_error("\n server: bad connection attempt from " + repr(self.addr) + ":\n") + self.running = False + self.server.stop() + self.close() + return False else: if self.server.context.verify_mode == ssl.CERT_REQUIRED: cert = self.sslconn.getpeercert() @@ -2204,10 +2260,10 @@ else: connect to it with a wrong client certificate fails. """ certfile = os.path.join(os.path.dirname(__file__) or os.curdir, - "wrongcert.pem") - server = ThreadedEchoServer(CERTFILE, + "keycert.pem") + server = ThreadedEchoServer(SIGNED_CERTFILE, certreqs=ssl.CERT_REQUIRED, - cacerts=CERTFILE, chatty=False, + cacerts=SIGNING_CA, chatty=False, connectionchatty=False) with server, \ closing(socket.socket()) as sock, \ @@ -2638,6 +2694,7 @@ else: self.assertEqual(s.read(-1, buffer), len(data)) self.assertEqual(buffer, data) + self.assertRaises(NotImplementedError, s.dup) s.write(b"over\n") self.assertRaises(ValueError, s.recv, -1) @@ -2734,7 +2791,7 @@ else: # Block on the accept and wait on the connection to close. evt.set() remote[0], peer[0] = server.accept() - remote[0].recv(1) + remote[0].send(remote[0].recv(4)) t = threading.Thread(target=serve) t.start() @@ -2742,6 +2799,8 @@ else: evt.wait() client = context.wrap_socket(socket.socket()) client.connect((host, port)) + client.send(b'data') + client.recv() client_addr = client.getsockname() client.close() t.join() @@ -2765,19 +2824,24 @@ else: sock.do_handshake() self.assertEqual(cm.exception.errno, errno.ENOTCONN) - def test_default_ciphers(self): - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - try: - # Force a set of weak ciphers on our client context - context.set_ciphers("DES") - except ssl.SSLError: - self.skipTest("no DES cipher available") - with ThreadedEchoServer(CERTFILE, - ssl_version=ssl.PROTOCOL_SSLv23, - chatty=False) as server: - with closing(context.wrap_socket(socket.socket())) as s: - with self.assertRaises(ssl.SSLError): - s.connect((HOST, server.port)) + def test_no_shared_ciphers(self): + server_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + server_context.load_cert_chain(SIGNED_CERTFILE) + client_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + client_context.verify_mode = ssl.CERT_REQUIRED + client_context.check_hostname = True + + # OpenSSL enables all TLS 1.3 ciphers, enforce TLS 1.2 for test + client_context.options |= ssl.OP_NO_TLSv1_3 + # Force different suites on client and master + client_context.set_ciphers("AES128") + server_context.set_ciphers("AES256") + with ThreadedEchoServer(context=server_context) as server: + s = client_context.wrap_socket( + socket.socket(), + server_hostname="localhost") + with self.assertRaises(ssl.SSLError): + s.connect((HOST, server.port)) self.assertIn("no shared cipher", str(server.conn_errors[0])) def test_version_basic(self): @@ -2805,12 +2869,13 @@ else: ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_TLSv1_2 ) with ThreadedEchoServer(context=context) as server: - with context.wrap_socket(socket.socket()) as s: + s = context.wrap_socket(socket.socket()) + with closing(s): s.connect((HOST, server.port)) self.assertIn(s.cipher()[0], [ - 'TLS13-AES-256-GCM-SHA384', - 'TLS13-CHACHA20-POLY1305-SHA256', - 'TLS13-AES-128-GCM-SHA256', + 'TLS_AES_256_GCM_SHA384', + 'TLS_CHACHA20_POLY1305_SHA256', + 'TLS_AES_128_GCM_SHA256', ]) @unittest.skipUnless(ssl.HAS_ECDH, "test requires ECDH-enabled OpenSSL") @@ -2819,6 +2884,9 @@ else: # should be enabled by default on SSL contexts. context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.load_cert_chain(CERTFILE) + # TLSv1.3 defaults to PFS key agreement and no longer has KEA in + # cipher name. + context.options |= ssl.OP_NO_TLSv1_3 # Prior to OpenSSL 1.0.0, ECDH ciphers have to be enabled # explicitly using the 'ECCdraft' cipher alias. Otherwise, # our default cipher list should prefer ECDH-based ciphers diff --git a/Lib/test/test_str.py b/Lib/test/test_str.py index 8b306f4..73ed542 100644 --- a/Lib/test/test_str.py +++ b/Lib/test/test_str.py @@ -474,8 +474,106 @@ class StrTest( self.assertEqual('lhs %% %r' % SubclassedStr('rhs'), "Success, self.__rmod__('lhs %% %r') was called") + +class CAPITest(unittest.TestCase): + + # Test PyString_FromFormat() + def test_from_format(self): + ctypes = test_support.import_module('ctypes') + _testcapi = test_support.import_module('_testcapi') + from ctypes import pythonapi, py_object + from ctypes import ( + c_int, c_uint, + c_long, c_ulong, + c_size_t, c_ssize_t, + c_char_p) + + PyString_FromFormat = pythonapi.PyString_FromFormat + PyString_FromFormat.restype = py_object + + # basic tests + self.assertEqual(PyString_FromFormat(b'format'), + b'format') + self.assertEqual(PyString_FromFormat(b'Hello %s !', b'world'), + b'Hello world !') + + # test formatters + self.assertEqual(PyString_FromFormat(b'c=%c', c_int(0)), + b'c=\0') + self.assertEqual(PyString_FromFormat(b'c=%c', c_int(ord('@'))), + b'c=@') + self.assertEqual(PyString_FromFormat(b'c=%c', c_int(255)), + b'c=\xff') + self.assertEqual(PyString_FromFormat(b'd=%d ld=%ld zd=%zd', + c_int(1), c_long(2), + c_size_t(3)), + b'd=1 ld=2 zd=3') + self.assertEqual(PyString_FromFormat(b'd=%d ld=%ld zd=%zd', + c_int(-1), c_long(-2), + c_size_t(-3)), + b'd=-1 ld=-2 zd=-3') + self.assertEqual(PyString_FromFormat(b'u=%u lu=%lu zu=%zu', + c_uint(123), c_ulong(456), + c_size_t(789)), + b'u=123 lu=456 zu=789') + self.assertEqual(PyString_FromFormat(b'i=%i', c_int(123)), + b'i=123') + self.assertEqual(PyString_FromFormat(b'i=%i', c_int(-123)), + b'i=-123') + self.assertEqual(PyString_FromFormat(b'x=%x', c_int(0xabc)), + b'x=abc') + + self.assertEqual(PyString_FromFormat(b's=%s', c_char_p(b'cstr')), + b's=cstr') + + # test minimum and maximum integer values + size_max = c_size_t(-1).value + for formatstr, ctypes_type, value, py_formatter in ( + (b'%d', c_int, _testcapi.INT_MIN, str), + (b'%d', c_int, _testcapi.INT_MAX, str), + (b'%ld', c_long, _testcapi.LONG_MIN, str), + (b'%ld', c_long, _testcapi.LONG_MAX, str), + (b'%lu', c_ulong, _testcapi.ULONG_MAX, str), + (b'%zd', c_ssize_t, _testcapi.PY_SSIZE_T_MIN, str), + (b'%zd', c_ssize_t, _testcapi.PY_SSIZE_T_MAX, str), + (b'%zu', c_size_t, size_max, str), + ): + self.assertEqual(PyString_FromFormat(formatstr, ctypes_type(value)), + py_formatter(value).encode('ascii')), + + # width and precision (width is currently ignored) + self.assertEqual(PyString_FromFormat(b'%5s', b'a'), + b'a') + self.assertEqual(PyString_FromFormat(b'%.3s', b'abcdef'), + b'abc') + + # '%%' formatter + self.assertEqual(PyString_FromFormat(b'%%'), + b'%') + self.assertEqual(PyString_FromFormat(b'[%%]'), + b'[%]') + self.assertEqual(PyString_FromFormat(b'%%%c', c_int(ord('_'))), + b'%_') + self.assertEqual(PyString_FromFormat(b'%%s'), + b'%s') + + # Invalid formats and partial formatting + self.assertEqual(PyString_FromFormat(b'%'), b'%') + self.assertEqual(PyString_FromFormat(b'x=%i y=%', c_int(2), c_int(3)), + b'x=2 y=%') + + self.assertEqual(PyString_FromFormat(b'%c', c_int(-1)), b'\xff') + self.assertEqual(PyString_FromFormat(b'%c', c_int(256)), b'\0') + + # Issue #33817: empty strings + self.assertEqual(PyString_FromFormat(b''), + b'') + self.assertEqual(PyString_FromFormat(b'%s', b''), + b'') + + def test_main(): - test_support.run_unittest(StrTest) + test_support.run_unittest(StrTest, CAPITest) if __name__ == "__main__": test_main() diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py index 3d24941..a7af85a 100644 --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -428,7 +428,7 @@ class CalculationTests(unittest.TestCase): self.assertTrue(result.tm_year == self.time_tuple.tm_year and result.tm_mon == self.time_tuple.tm_mon and result.tm_mday == self.time_tuple.tm_mday, - "Calculation of Gregorian date failed;" + "Calculation of Gregorian date failed; " "%s-%s-%s != %s-%s-%s" % (result.tm_year, result.tm_mon, result.tm_mday, self.time_tuple.tm_year, self.time_tuple.tm_mon, @@ -440,7 +440,7 @@ class CalculationTests(unittest.TestCase): result = _strptime._strptime_time(time.strftime(format_string, self.time_tuple), format_string) self.assertTrue(result.tm_wday == self.time_tuple.tm_wday, - "Calculation of day of the week failed;" + "Calculation of day of the week failed; " "%s != %s" % (result.tm_wday, self.time_tuple.tm_wday)) def test_week_of_year_and_day_of_week_calculation(self): @@ -580,7 +580,7 @@ class CacheTests(unittest.TestCase): finally: locale.setlocale(locale.LC_TIME, locale_info) - @support.run_with_tz('STD-1DST') + @support.run_with_tz('STD-1DST,M4.1.0,M10.1.0') def test_TimeRE_recreation_timezone(self): # The TimeRE instance should be recreated upon changing the timezone. oldtzname = time.tzname diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index ee2383b..0c215f7 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -843,8 +843,11 @@ class _SuppressCoreFiles(object): kw = {stream: subprocess.PIPE} with subprocess.Popen(args, **kw) as process: signal.alarm(1) - # communicate() will be interrupted by SIGALRM - process.communicate() + try: + # communicate() will be interrupted by SIGALRM + process.communicate() + finally: + signal.alarm(0) @unittest.skipIf(mswindows, "POSIX specific tests") diff --git a/Lib/test/test_sundry.py b/Lib/test/test_sundry.py index 8fe8995..d4a97c8 100644 --- a/Lib/test/test_sundry.py +++ b/Lib/test/test_sundry.py @@ -10,7 +10,6 @@ class TestUntestedModules(unittest.TestCase): with test_support.check_warnings(quiet=True): import CGIHTTPServer import audiodev - import bdb import cgitb import code import compileall diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py index d8737af..b6695fe 100644 --- a/Lib/test/test_sys_settrace.py +++ b/Lib/test/test_sys_settrace.py @@ -1053,7 +1053,7 @@ output.append(4) "can only jump from a 'line' trace event")) def test_no_jump_from_exception_event(output): output.append(1) - 1 / 0 + 1 // 0 @jump_test(3, 2, [2], event='return', error=(ValueError, "can't jump from a yield statement")) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index f0c9877..84c4cea 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -661,6 +661,43 @@ class TclTest(unittest.TestCase): expected = {'a': (1, 2, 3), 'something': 'foo', 'status': ''} self.assertEqual(splitdict(tcl, arg), expected) + def test_join(self): + join = tkinter._join + tcl = self.interp.tk + def unpack(s): + return tcl.call('lindex', s, 0) + def check(value): + self.assertEqual(unpack(join([value])), value) + self.assertEqual(unpack(join([value, 0])), value) + self.assertEqual(unpack(unpack(join([[value]]))), value) + self.assertEqual(unpack(unpack(join([[value, 0]]))), value) + self.assertEqual(unpack(unpack(join([[value], 0]))), value) + self.assertEqual(unpack(unpack(join([[value, 0], 0]))), value) + check('') + check('spam') + check('sp am') + check('sp\tam') + check('sp\nam') + check(' \t\n') + check('{spam}') + check('{sp am}') + check('"spam"') + check('"sp am"') + check('{"spam"}') + check('"{spam}"') + check('sp\\am') + check('"sp\\am"') + check('"{}" "{}"') + check('"\\') + check('"{') + check('"}') + check('\n\\') + check('\n{') + check('\n}') + check('\\\n') + check('{\n') + check('}\n') + character_size = 4 if sys.maxunicode > 0xFFFF else 2 diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 5c111a2..2efb836 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -821,6 +821,7 @@ class test_NamedTemporaryFile(TC): old_fdopen = os.fdopen closed = [] def close(fd): + old_close(fd) closed.append(fd) def fdopen(*args): raise ValueError() diff --git a/Lib/test/test_test_support.py b/Lib/test/test_test_support.py index 421a9ef..f9192a7 100644 --- a/Lib/test/test_test_support.py +++ b/Lib/test/test_test_support.py @@ -267,7 +267,7 @@ class TestSupport(unittest.TestCase): with support.temp_cwd(name=TESTFN): self.assertEqual(os.path.basename(os.getcwd()), TESTFN) self.assertFalse(os.path.exists(TESTFN)) - self.assertTrue(os.path.basename(os.getcwd()), here) + self.assertEqual(os.getcwd(), here) def test_temp_cwd__name_none(self): @@ -417,6 +417,17 @@ class TestSupport(unittest.TestCase): self.assertTrue(support.match_test(test_access)) self.assertFalse(support.match_test(test_chdir)) + def test_fd_count(self): + # We cannot test the absolute value of fd_count(): on old Linux + # kernel or glibc versions, os.urandom() keeps a FD open on + # /dev/urandom device and Python has 4 FD opens instead of 3. + start = support.fd_count() + fd = os.open(__file__, os.O_RDONLY) + try: + more = support.fd_count() + finally: + os.close(fd) + self.assertEqual(more - start, 1) # XXX -follows a list of untested API # make_legacy_pyc diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py index c8caa5d..93690a6 100644 --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -1,8 +1,8 @@ import os import unittest import random -from test import test_support -thread = test_support.import_module('thread') +from test import support +thread = support.import_module('thread') import time import sys import weakref @@ -17,7 +17,7 @@ _print_mutex = thread.allocate_lock() def verbose_print(arg): """Helper function for printing out debugging output.""" - if test_support.verbose: + if support.verbose: with _print_mutex: print arg @@ -34,8 +34,8 @@ class BasicThreadTest(unittest.TestCase): self.running = 0 self.next_ident = 0 - key = test_support.threading_setup() - self.addCleanup(test_support.threading_cleanup, *key) + key = support.threading_setup() + self.addCleanup(support.threading_cleanup, *key) class ThreadRunningTests(BasicThreadTest): @@ -60,12 +60,13 @@ class ThreadRunningTests(BasicThreadTest): self.done_mutex.release() def test_starting_threads(self): - # Basic test for thread creation. - for i in range(NUMTASKS): - self.newtask() - verbose_print("waiting for tasks to complete...") - self.done_mutex.acquire() - verbose_print("all tasks done") + with support.wait_threads_exit(): + # Basic test for thread creation. + for i in range(NUMTASKS): + self.newtask() + verbose_print("waiting for tasks to complete...") + self.done_mutex.acquire() + verbose_print("all tasks done") def test_stack_size(self): # Various stack size tests. @@ -95,12 +96,13 @@ class ThreadRunningTests(BasicThreadTest): verbose_print("trying stack_size = (%d)" % tss) self.next_ident = 0 self.created = 0 - for i in range(NUMTASKS): - self.newtask() + with support.wait_threads_exit(): + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") thread.stack_size(0) @@ -110,25 +112,28 @@ class ThreadRunningTests(BasicThreadTest): mut = thread.allocate_lock() mut.acquire() started = [] + def task(): started.append(None) mut.acquire() mut.release() - thread.start_new_thread(task, ()) - while not started: - time.sleep(0.01) - self.assertEqual(thread._count(), orig + 1) - # Allow the task to finish. - mut.release() - # The only reliable way to be sure that the thread ended from the - # interpreter's point of view is to wait for the function object to be - # destroyed. - done = [] - wr = weakref.ref(task, lambda _: done.append(None)) - del task - while not done: - time.sleep(0.01) - self.assertEqual(thread._count(), orig) + + with support.wait_threads_exit(): + thread.start_new_thread(task, ()) + while not started: + time.sleep(0.01) + self.assertEqual(thread._count(), orig + 1) + # Allow the task to finish. + mut.release() + # The only reliable way to be sure that the thread ended from the + # interpreter's point of view is to wait for the function object to be + # destroyed. + done = [] + wr = weakref.ref(task, lambda _: done.append(None)) + del task + while not done: + time.sleep(0.01) + self.assertEqual(thread._count(), orig) def test_save_exception_state_on_error(self): # See issue #14474 @@ -143,14 +148,13 @@ class ThreadRunningTests(BasicThreadTest): real_write(self, *args) c = thread._count() started = thread.allocate_lock() - with test_support.captured_output("stderr") as stderr: + with support.captured_output("stderr") as stderr: real_write = stderr.write stderr.write = mywrite started.acquire() - thread.start_new_thread(task, ()) - started.acquire() - while thread._count() > c: - time.sleep(0.01) + with support.wait_threads_exit(): + thread.start_new_thread(task, ()) + started.acquire() self.assertIn("Traceback", stderr.getvalue()) @@ -182,13 +186,14 @@ class Barrier: class BarrierTest(BasicThreadTest): def test_barrier(self): - self.bar = Barrier(NUMTASKS) - self.running = NUMTASKS - for i in range(NUMTASKS): - thread.start_new_thread(self.task2, (i,)) - verbose_print("waiting for tasks to end") - self.done_mutex.acquire() - verbose_print("tasks done") + with support.wait_threads_exit(): + self.bar = Barrier(NUMTASKS) + self.running = NUMTASKS + for i in range(NUMTASKS): + thread.start_new_thread(self.task2, (i,)) + verbose_print("waiting for tasks to end") + self.done_mutex.acquire() + verbose_print("tasks done") def task2(self, ident): for i in range(NUMTRIPS): @@ -226,8 +231,9 @@ class TestForkInThread(unittest.TestCase): @unittest.skipIf(sys.platform.startswith('win'), "This test is only appropriate for POSIX-like systems.") - @test_support.reap_threads + @support.reap_threads def test_forkinthread(self): + non_local = {'status': None} def thread1(): try: pid = os.fork() # fork in a thread @@ -246,11 +252,13 @@ class TestForkInThread(unittest.TestCase): else: # parent os.close(self.write_fd) pid, status = os.waitpid(pid, 0) - self.assertEqual(status, 0) + non_local['status'] = status - thread.start_new_thread(thread1, ()) - self.assertEqual(os.read(self.read_fd, 2), "OK", - "Unable to fork() in thread") + with support.wait_threads_exit(): + thread.start_new_thread(thread1, ()) + self.assertEqual(os.read(self.read_fd, 2), "OK", + "Unable to fork() in thread") + self.assertEqual(non_local['status'], 0) def tearDown(self): try: @@ -265,7 +273,7 @@ class TestForkInThread(unittest.TestCase): def test_main(): - test_support.run_unittest(ThreadRunningTests, BarrierTest, LockTests, + support.run_unittest(ThreadRunningTests, BarrierTest, LockTests, TestForkInThread) if __name__ == "__main__": diff --git a/Lib/test/test_threadsignals.py b/Lib/test/test_threadsignals.py index 2f7eb60..3d79fd5 100644 --- a/Lib/test/test_threadsignals.py +++ b/Lib/test/test_threadsignals.py @@ -52,9 +52,11 @@ class ThreadSignals(unittest.TestCase): # wait for it return. if signal_blackboard[signal.SIGUSR1]['tripped'] == 0 \ or signal_blackboard[signal.SIGUSR2]['tripped'] == 0: - signal.alarm(1) - signal.pause() - signal.alarm(0) + try: + signal.alarm(1) + signal.pause() + finally: + signal.alarm(0) self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped'], 1) self.assertEqual( signal_blackboard[signal.SIGUSR1]['tripped_by'], diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py index 57b3ef1..51e4fd6 100644 --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -416,12 +416,15 @@ class FixcidTests(unittest.TestCase): with open(os.path.join(test_support.TESTFN, "file.py"), "w") as file: file.write("xx = 'unaltered'\n") script = os.path.join(scriptsdir, "fixcid.py") - output = self.run_script(args=(test_support.TESTFN,)) + # ignore dbg() messages + with test_support.captured_stderr() as stderr: + output = self.run_script(args=(test_support.TESTFN,)) self.assertMultiLineEqual(output, "{}:\n" "1\n" '< int xx;\n' - '> int yy;\n'.format(c_filename) + '> int yy;\n'.format(c_filename), + "stderr: %s" % stderr.getvalue() ) def run_script(self, input="", args=("-",), substfile="xx yy\n"): diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index 560b84c..92476f6 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1824,6 +1824,12 @@ class CAPITest(unittest.TestCase): check_format(u'%s', b'%.%s', b'abc') + # Issue #33817: empty strings + check_format(u'', + b'') + check_format(u'', + b'%s', b'') + @test_support.cpython_only def test_encode_decimal(self): from _testcapi import unicode_encodedecimal diff --git a/Lib/test/test_unicodedata.py b/Lib/test/test_unicodedata.py index c30ecf4..11f2cda 100644 --- a/Lib/test/test_unicodedata.py +++ b/Lib/test/test_unicodedata.py @@ -204,6 +204,19 @@ class UnicodeFunctionsTest(UnicodeDatabaseTest): b = u'C\u0338' * 20 + u'\xC7' self.assertEqual(self.db.normalize('NFC', a), b) + def test_issue29456(self): + # Fix #29456 + u1176_str_a = u'\u1100\u1176\u11a8' + u1176_str_b = u'\u1100\u1176\u11a8' + u11a7_str_a = u'\u1100\u1175\u11a7' + u11a7_str_b = u'\uae30\u11a7' + u11c3_str_a = u'\u1100\u1175\u11c3' + u11c3_str_b = u'\uae30\u11c3' + self.assertEqual(self.db.normalize('NFC', u1176_str_a), u1176_str_b) + self.assertEqual(self.db.normalize('NFC', u11a7_str_a), u11a7_str_b) + self.assertEqual(self.db.normalize('NFC', u11c3_str_a), u11c3_str_b) + + def test_east_asian_width(self): eaw = self.db.east_asian_width self.assertRaises(TypeError, eaw, 'a') diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py index 89ee759..ee0b7fe 100644 --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -25,6 +25,13 @@ def _wrap_with_retry_thrice(func, exc): return _retry_thrice(func, exc, *args, **kwargs) return wrapped +# bpo-35411: FTP tests of test_urllib2net randomly fail +# with "425 Security: Bad IP connecting" on Travis CI +skip_ftp_test_on_travis = unittest.skipIf('TRAVIS' in os.environ, + 'bpo-35411: skip FTP test ' + 'on Travis CI') + + # Connecting to remote hosts is flaky. Make it more robust by retrying # the connection several times. _urlopen_with_retry = _wrap_with_retry_thrice(urllib2.urlopen, urllib2.URLError) @@ -100,6 +107,7 @@ class OtherNetworkTests(unittest.TestCase): # XXX The rest of these tests aren't very good -- they don't check much. # They do sometimes catch some major disasters, though. + @skip_ftp_test_on_travis def test_ftp(self): urls = [ 'ftp://www.pythontest.net/README', @@ -184,6 +192,7 @@ class OtherNetworkTests(unittest.TestCase): opener.open(request) self.assertEqual(request.get_header('User-agent'),'Test-Agent') + @unittest.skip('XXX: http://www.imdb.com is gone') def test_sites_no_connection_close(self): # Some sites do not send Connection: close header. # Verify that those work properly. (#issue12576) @@ -284,12 +293,14 @@ class TimeoutTest(unittest.TestCase): FTP_HOST = 'ftp://www.pythontest.net/' + @skip_ftp_test_on_travis def test_ftp_basic(self): self.assertIsNone(socket.getdefaulttimeout()) with test_support.transient_internet(self.FTP_HOST, timeout=None): u = _urlopen_with_retry(self.FTP_HOST) self.assertIsNone(u.fp.fp._sock.gettimeout()) + @skip_ftp_test_on_travis def test_ftp_default_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) with test_support.transient_internet(self.FTP_HOST): @@ -300,6 +311,7 @@ class TimeoutTest(unittest.TestCase): socket.setdefaulttimeout(None) self.assertEqual(u.fp.fp._sock.gettimeout(), 60) + @skip_ftp_test_on_travis def test_ftp_no_timeout(self): self.assertIsNone(socket.getdefaulttimeout(),) with test_support.transient_internet(self.FTP_HOST): @@ -310,6 +322,7 @@ class TimeoutTest(unittest.TestCase): socket.setdefaulttimeout(None) self.assertIsNone(u.fp.fp._sock.gettimeout()) + @skip_ftp_test_on_travis def test_ftp_timeout(self): with test_support.transient_internet(self.FTP_HOST): u = _urlopen_with_retry(self.FTP_HOST, timeout=60) diff --git a/Lib/test/test_uu.py b/Lib/test/test_uu.py index 51a4fbe..df41cbc 100644 --- a/Lib/test/test_uu.py +++ b/Lib/test/test_uu.py @@ -4,9 +4,10 @@ Nick Mathewson """ import unittest -from test import test_support +from test import test_support as support -import sys, os, uu, cStringIO +import cStringIO +import sys import uu plaintext = "The smooth-scaled python crept over the sleeping dog\n" @@ -108,114 +109,64 @@ class UUStdIOTest(unittest.TestCase): class UUFileTest(unittest.TestCase): - def _kill(self, f): - # close and remove file - try: - f.close() - except (SystemExit, KeyboardInterrupt): - raise - except: - pass - try: - os.unlink(f.name) - except (SystemExit, KeyboardInterrupt): - raise - except: - pass - def setUp(self): - self.tmpin = test_support.TESTFN + "i" - self.tmpout = test_support.TESTFN + "o" - - def tearDown(self): - del self.tmpin - del self.tmpout + self.tmpin = support.TESTFN + "i" + self.tmpout = support.TESTFN + "o" + self.addCleanup(support.unlink, self.tmpin) + self.addCleanup(support.unlink, self.tmpout) def test_encode(self): - fin = fout = None - try: - test_support.unlink(self.tmpin) - fin = open(self.tmpin, 'wb') + with open(self.tmpin, 'wb') as fin: fin.write(plaintext) - fin.close() - fin = open(self.tmpin, 'rb') - fout = open(self.tmpout, 'w') - uu.encode(fin, fout, self.tmpin, mode=0644) - fin.close() - fout.close() + with open(self.tmpin, 'rb') as fin: + with open(self.tmpout, 'w') as fout: + uu.encode(fin, fout, self.tmpin, mode=0o644) - fout = open(self.tmpout, 'r') + with open(self.tmpout, 'r') as fout: s = fout.read() - fout.close() - self.assertEqual(s, encodedtextwrapped % (0644, self.tmpin)) + self.assertEqual(s, encodedtextwrapped % (0o644, self.tmpin)) - # in_file and out_file as filenames - uu.encode(self.tmpin, self.tmpout, self.tmpin, mode=0644) - fout = open(self.tmpout, 'r') + # in_file and out_file as filenames + uu.encode(self.tmpin, self.tmpout, self.tmpin, mode=0o644) + with open(self.tmpout, 'r') as fout: s = fout.read() - fout.close() - self.assertEqual(s, encodedtextwrapped % (0644, self.tmpin)) - - finally: - self._kill(fin) - self._kill(fout) + self.assertEqual(s, encodedtextwrapped % (0o644, self.tmpin)) def test_decode(self): - f = None - try: - test_support.unlink(self.tmpin) - f = open(self.tmpin, 'w') - f.write(encodedtextwrapped % (0644, self.tmpout)) - f.close() + with open(self.tmpin, 'w') as f: + f.write(encodedtextwrapped % (0o644, self.tmpout)) - f = open(self.tmpin, 'r') + with open(self.tmpin, 'r') as f: uu.decode(f) - f.close() - f = open(self.tmpout, 'r') + with open(self.tmpout, 'r') as f: s = f.read() - f.close() - self.assertEqual(s, plaintext) - # XXX is there an xp way to verify the mode? - finally: - self._kill(f) + self.assertEqual(s, plaintext) + # XXX is there an xp way to verify the mode? def test_decode_filename(self): - f = None - try: - test_support.unlink(self.tmpin) - f = open(self.tmpin, 'w') - f.write(encodedtextwrapped % (0644, self.tmpout)) - f.close() + with open(self.tmpin, 'w') as f: + f.write(encodedtextwrapped % (0o644, self.tmpout)) - uu.decode(self.tmpin) + uu.decode(self.tmpin) - f = open(self.tmpout, 'r') + with open(self.tmpout, 'r') as f: s = f.read() - f.close() - self.assertEqual(s, plaintext) - finally: - self._kill(f) + self.assertEqual(s, plaintext) def test_decodetwice(self): # Verify that decode() will refuse to overwrite an existing file - f = None - try: - f = cStringIO.StringIO(encodedtextwrapped % (0644, self.tmpout)) - - f = open(self.tmpin, 'r') + with open(self.tmpin, 'wb') as f: + f.write(encodedtextwrapped % (0o644, self.tmpout)) + with open(self.tmpin, 'r') as f: uu.decode(f) - f.close() - f = open(self.tmpin, 'r') + with open(self.tmpin, 'r') as f: self.assertRaises(uu.Error, uu.decode, f) - f.close() - finally: - self._kill(f) def test_main(): - test_support.run_unittest(UUTest, UUStdIOTest, UUFileTest) + support.run_unittest(UUTest, UUStdIOTest, UUFileTest) if __name__=="__main__": test_main() diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index 3749564..6cd2c39 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -287,6 +287,39 @@ class TestUUID(unittest.TestCase): node2 = uuid.getnode() self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2)) + # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers + # need not necessarily be 48 bits (e.g., EUI-64). + def test_uuid1_eui64(self): + # Confirm that uuid.getnode ignores hardware addresses larger than 48 + # bits. Mock out each platform's *_getnode helper functions to return + # something just larger than 48 bits to test. This will cause + # uuid.getnode to fall back on uuid._random_getnode, which will + # generate a valid value. + too_large_getter = lambda: 1 << 48 + + uuid_real__node = uuid._node + uuid_real__NODE_GETTERS_WIN32 = uuid._NODE_GETTERS_WIN32 + uuid_real__NODE_GETTERS_UNIX = uuid._NODE_GETTERS_UNIX + uuid._node = None + uuid._NODE_GETTERS_WIN32 = [too_large_getter] + uuid._NODE_GETTERS_UNIX = [too_large_getter] + try: + node = uuid.getnode() + finally: + uuid._node = uuid_real__node + uuid._NODE_GETTERS_WIN32 = uuid_real__NODE_GETTERS_WIN32 + uuid._NODE_GETTERS_UNIX = uuid_real__NODE_GETTERS_UNIX + + self.assertTrue(0 < node < (1 << 48), '%012x' % node) + + # Confirm that uuid1 can use the generated node, i.e., the that + # uuid.getnode fell back on uuid._random_getnode() rather than using + # the value from too_large_getter above. + try: + uuid.uuid1(node=node) + except ValueError as e: + self.fail('uuid1 was given an invalid node ID') + @unittest.skipUnless(importable('ctypes'), 'requires ctypes') def test_uuid1(self): equal = self.assertEqual diff --git a/Lib/test/test_warnings.py b/Lib/test/test_warnings.py index ae081ee..a9568f6 100644 --- a/Lib/test/test_warnings.py +++ b/Lib/test/test_warnings.py @@ -107,10 +107,14 @@ class FilterTests(object): self.module.resetwarnings() self.module.filterwarnings("always", category=UserWarning) message = "FilterTests.test_always" - self.module.warn(message, UserWarning) - self.assertTrue(message, w[-1].message) - self.module.warn(message, UserWarning) - self.assertTrue(w[-1].message, message) + def f(): + self.module.warn(message, UserWarning) + f() + self.assertEqual(len(w), 1) + self.assertEqual(w[-1].message.args[0], message) + f() + self.assertEqual(len(w), 2) + self.assertEqual(w[-1].message.args[0], message) def test_default(self): with original_warnings.catch_warnings(record=True, diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index 97d9e8f..36b3be6 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -854,13 +854,9 @@ class GzipServerTestCase(BaseServerTestCase): class ServerProxyTestCase(unittest.TestCase): def setUp(self): unittest.TestCase.setUp(self) - if threading: - self.url = URL - else: - # Without threading, http_server() and http_multi_server() will not - # be executed and URL is still equal to None. 'http://' is a just - # enough to choose the scheme (HTTP) - self.url = 'http://' + # Actual value of the URL doesn't matter if it is a string in + # the correct format. + self.url = 'http://fake.localhost' def test_close(self): p = xmlrpclib.ServerProxy(self.url) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 9c63aeb..4e545f1 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -812,6 +812,20 @@ class TestZip64InSmallFiles(unittest.TestCase): self.assertEqual(content, "%d" % (i**3 % 57)) zipf2.close() + def test_append(self): + # Test that appending to the Zip64 archive doesn't change + # extra fields of existing entries. + with zipfile.ZipFile(TESTFN2, "w", allowZip64=True) as zipfp: + zipfp.writestr("strfile", self.data) + with zipfile.ZipFile(TESTFN2, "r", allowZip64=True) as zipfp: + zinfo = zipfp.getinfo("strfile") + extra = zinfo.extra + with zipfile.ZipFile(TESTFN2, "a", allowZip64=True) as zipfp: + zipfp.writestr("strfile2", self.data) + with zipfile.ZipFile(TESTFN2, "r", allowZip64=True) as zipfp: + zinfo = zipfp.getinfo("strfile") + self.assertEqual(zinfo.extra, extra) + def tearDown(self): zipfile.ZIP64_LIMIT = self._limit zipfile.ZIP_FILECOUNT_LIMIT = self._filecount_limit diff --git a/Lib/test/wrongcert.pem b/Lib/test/wrongcert.pem deleted file mode 100644 index 5f92f9b..0000000 --- a/Lib/test/wrongcert.pem +++ /dev/null @@ -1,32 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIICXAIBAAKBgQC89ZNxjTgWgq7Z1g0tJ65w+k7lNAj5IgjLb155UkUrz0XsHDnH -FlbsVUg2Xtk6+bo2UEYIzN7cIm5ImpmyW/2z0J1IDVDlvR2xJ659xrE0v5c2cB6T -f9lnNTwpSoeK24Nd7Jwq4j9vk95fLrdqsBq0/KVlsCXeixS/CaqqduXfvwIDAQAB -AoGAQFko4uyCgzfxr4Ezb4Mp5pN3Npqny5+Jey3r8EjSAX9Ogn+CNYgoBcdtFgbq -1yif/0sK7ohGBJU9FUCAwrqNBI9ZHB6rcy7dx+gULOmRBGckln1o5S1+smVdmOsW -7zUVLBVByKuNWqTYFlzfVd6s4iiXtAE2iHn3GCyYdlICwrECQQDhMQVxHd3EFbzg -SFmJBTARlZ2GKA3c1g/h9/XbkEPQ9/RwI3vnjJ2RaSnjlfoLl8TOcf0uOGbOEyFe -19RvCLXjAkEA1s+UE5ziF+YVkW3WolDCQ2kQ5WG9+ccfNebfh6b67B7Ln5iG0Sbg -ky9cjsO3jbMJQtlzAQnH1850oRD5Gi51dQJAIbHCDLDZU9Ok1TI+I2BhVuA6F666 -lEZ7TeZaJSYq34OaUYUdrwG9OdqwZ9sy9LUav4ESzu2lhEQchCJrKMn23QJAReqs -ZLHUeTjfXkVk7dHhWPWSlUZ6AhmIlA/AQ7Payg2/8wM/JkZEJEPvGVykms9iPUrv -frADRr+hAGe43IewnQJBAJWKZllPgKuEBPwoEldHNS8nRu61D7HzxEzQ2xnfj+Nk -2fgf1MAzzTRsikfGENhVsVWeqOcijWb6g5gsyCmlRpc= ------END RSA PRIVATE KEY----- ------BEGIN CERTIFICATE----- -MIICsDCCAhmgAwIBAgIJAOqYOYFJfEEoMA0GCSqGSIb3DQEBBQUAMEUxCzAJBgNV -BAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0YXRlMSEwHwYDVQQKExhJbnRlcm5ldCBX -aWRnaXRzIFB0eSBMdGQwHhcNMDgwNjI2MTgxNTUyWhcNMDkwNjI2MTgxNTUyWjBF -MQswCQYDVQQGEwJBVTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50 -ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB -gQC89ZNxjTgWgq7Z1g0tJ65w+k7lNAj5IgjLb155UkUrz0XsHDnHFlbsVUg2Xtk6 -+bo2UEYIzN7cIm5ImpmyW/2z0J1IDVDlvR2xJ659xrE0v5c2cB6Tf9lnNTwpSoeK -24Nd7Jwq4j9vk95fLrdqsBq0/KVlsCXeixS/CaqqduXfvwIDAQABo4GnMIGkMB0G -A1UdDgQWBBTctMtI3EO9OjLI0x9Zo2ifkwIiNjB1BgNVHSMEbjBsgBTctMtI3EO9 -OjLI0x9Zo2ifkwIiNqFJpEcwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgTClNvbWUt -U3RhdGUxITAfBgNVBAoTGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZIIJAOqYOYFJ -fEEoMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAQwa7jya/DfhaDn7E -usPkpgIX8WCL2B1SqnRTXEZfBPPVq/cUmFGyEVRVATySRuMwi8PXbVcOhXXuocA+ -43W+iIsD9pXapCZhhOerCq18TC1dWK98vLUsoK8PMjB6e5H/O8bqojv0EeC+fyCw -eSHj5jpC8iZKjCHBn+mAi4cQ514= ------END CERTIFICATE----- diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py deleted file mode 100644 index 5b0e29c..0000000 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ /dev/null @@ -1,1872 +0,0 @@ -import unittest -import tkinter -from tkinter import ttk, TclError -from test.support import requires -import sys - -from tkinter.test.test_ttk.test_functions import MockTclObj -from tkinter.test.support import (AbstractTkTest, tcl_version, get_tk_patchlevel, - simulate_mouse_click) -from tkinter.test.widget_tests import (add_standard_options, noconv, - AbstractWidgetTest, StandardOptionsTests, IntegerSizeTests, PixelSizeTests, - setUpModule) - -requires('gui') - - -class StandardTtkOptionsTests(StandardOptionsTests): - - def test_class(self): - widget = self.create() - self.assertEqual(widget['class'], '') - errmsg='attempt to change read-only option' - if get_tk_patchlevel() < (8, 6, 0, 'beta', 3): - errmsg='Attempt to change read-only option' - self.checkInvalidParam(widget, 'class', 'Foo', errmsg=errmsg) - widget2 = self.create(class_='Foo') - self.assertEqual(widget2['class'], 'Foo') - - def test_padding(self): - widget = self.create() - self.checkParam(widget, 'padding', 0, expected=('0',)) - self.checkParam(widget, 'padding', 5, expected=('5',)) - self.checkParam(widget, 'padding', (5, 6), expected=('5', '6')) - self.checkParam(widget, 'padding', (5, 6, 7), - expected=('5', '6', '7')) - self.checkParam(widget, 'padding', (5, 6, 7, 8), - expected=('5', '6', '7', '8')) - self.checkParam(widget, 'padding', ('5p', '6p', '7p', '8p')) - self.checkParam(widget, 'padding', (), expected='') - - def test_style(self): - widget = self.create() - self.assertEqual(widget['style'], '') - errmsg = 'Layout Foo not found' - if hasattr(self, 'default_orient'): - errmsg = ('Layout %s.Foo not found' % - getattr(self, 'default_orient').title()) - self.checkInvalidParam(widget, 'style', 'Foo', - errmsg=errmsg) - widget2 = self.create(class_='Foo') - self.assertEqual(widget2['class'], 'Foo') - # XXX - pass - - -class WidgetTest(AbstractTkTest, unittest.TestCase): - """Tests methods available in every ttk widget.""" - - def setUp(self): - super().setUp() - self.widget = ttk.Button(self.root, width=0, text="Text") - self.widget.pack() - self.widget.wait_visibility() - - - def test_identify(self): - self.widget.update_idletasks() - self.assertEqual(self.widget.identify( - int(self.widget.winfo_width() / 2), - int(self.widget.winfo_height() / 2) - ), "label") - self.assertEqual(self.widget.identify(-1, -1), "") - - self.assertRaises(tkinter.TclError, self.widget.identify, None, 5) - self.assertRaises(tkinter.TclError, self.widget.identify, 5, None) - self.assertRaises(tkinter.TclError, self.widget.identify, 5, '') - - - def test_widget_state(self): - # XXX not sure about the portability of all these tests - self.assertEqual(self.widget.state(), ()) - self.assertEqual(self.widget.instate(['!disabled']), True) - - # changing from !disabled to disabled - self.assertEqual(self.widget.state(['disabled']), ('!disabled', )) - # no state change - self.assertEqual(self.widget.state(['disabled']), ()) - # change back to !disable but also active - self.assertEqual(self.widget.state(['!disabled', 'active']), - ('!active', 'disabled')) - # no state changes, again - self.assertEqual(self.widget.state(['!disabled', 'active']), ()) - self.assertEqual(self.widget.state(['active', '!disabled']), ()) - - def test_cb(arg1, **kw): - return arg1, kw - self.assertEqual(self.widget.instate(['!disabled'], - test_cb, "hi", **{"msg": "there"}), - ('hi', {'msg': 'there'})) - - # attempt to set invalid statespec - currstate = self.widget.state() - self.assertRaises(tkinter.TclError, self.widget.instate, - ['badstate']) - self.assertRaises(tkinter.TclError, self.widget.instate, - ['disabled', 'badstate']) - # verify that widget didn't change its state - self.assertEqual(currstate, self.widget.state()) - - # ensuring that passing None as state doesn't modify current state - self.widget.state(['active', '!disabled']) - self.assertEqual(self.widget.state(), ('active', )) - - -class AbstractToplevelTest(AbstractWidgetTest, PixelSizeTests): - _conv_pixels = noconv - - -@add_standard_options(StandardTtkOptionsTests) -class FrameTest(AbstractToplevelTest, unittest.TestCase): - OPTIONS = ( - 'borderwidth', 'class', 'cursor', 'height', - 'padding', 'relief', 'style', 'takefocus', - 'width', - ) - - def create(self, **kwargs): - return ttk.Frame(self.root, **kwargs) - - -@add_standard_options(StandardTtkOptionsTests) -class LabelFrameTest(AbstractToplevelTest, unittest.TestCase): - OPTIONS = ( - 'borderwidth', 'class', 'cursor', 'height', - 'labelanchor', 'labelwidget', - 'padding', 'relief', 'style', 'takefocus', - 'text', 'underline', 'width', - ) - - def create(self, **kwargs): - return ttk.LabelFrame(self.root, **kwargs) - - def test_labelanchor(self): - widget = self.create() - self.checkEnumParam(widget, 'labelanchor', - 'e', 'en', 'es', 'n', 'ne', 'nw', 's', 'se', 'sw', 'w', 'wn', 'ws', - errmsg='Bad label anchor specification {}') - self.checkInvalidParam(widget, 'labelanchor', 'center') - - def test_labelwidget(self): - widget = self.create() - label = ttk.Label(self.root, text='Mupp', name='foo') - self.checkParam(widget, 'labelwidget', label, expected='.foo') - label.destroy() - - -class AbstractLabelTest(AbstractWidgetTest): - - def checkImageParam(self, widget, name): - image = tkinter.PhotoImage(master=self.root, name='image1') - image2 = tkinter.PhotoImage(master=self.root, name='image2') - self.checkParam(widget, name, image, expected=('image1',)) - self.checkParam(widget, name, 'image1', expected=('image1',)) - self.checkParam(widget, name, (image,), expected=('image1',)) - self.checkParam(widget, name, (image, 'active', image2), - expected=('image1', 'active', 'image2')) - self.checkParam(widget, name, 'image1 active image2', - expected=('image1', 'active', 'image2')) - self.checkInvalidParam(widget, name, 'spam', - errmsg='image "spam" doesn\'t exist') - - def test_compound(self): - widget = self.create() - self.checkEnumParam(widget, 'compound', - 'none', 'text', 'image', 'center', - 'top', 'bottom', 'left', 'right') - - def test_state(self): - widget = self.create() - self.checkParams(widget, 'state', 'active', 'disabled', 'normal') - - def test_width(self): - widget = self.create() - self.checkParams(widget, 'width', 402, -402, 0) - - -@add_standard_options(StandardTtkOptionsTests) -class LabelTest(AbstractLabelTest, unittest.TestCase): - OPTIONS = ( - 'anchor', 'background', 'borderwidth', - 'class', 'compound', 'cursor', 'font', 'foreground', - 'image', 'justify', 'padding', 'relief', 'state', 'style', - 'takefocus', 'text', 'textvariable', - 'underline', 'width', 'wraplength', - ) - _conv_pixels = noconv - - def create(self, **kwargs): - return ttk.Label(self.root, **kwargs) - - def test_font(self): - widget = self.create() - self.checkParam(widget, 'font', - '-Adobe-Helvetica-Medium-R-Normal--*-120-*-*-*-*-*-*') - - -@add_standard_options(StandardTtkOptionsTests) -class ButtonTest(AbstractLabelTest, unittest.TestCase): - OPTIONS = ( - 'class', 'command', 'compound', 'cursor', 'default', - 'image', 'padding', 'state', 'style', - 'takefocus', 'text', 'textvariable', - 'underline', 'width', - ) - - def create(self, **kwargs): - return ttk.Button(self.root, **kwargs) - - def test_default(self): - widget = self.create() - self.checkEnumParam(widget, 'default', 'normal', 'active', 'disabled') - - def test_invoke(self): - success = [] - btn = ttk.Button(self.root, command=lambda: success.append(1)) - btn.invoke() - self.assertTrue(success) - - -@add_standard_options(StandardTtkOptionsTests) -class CheckbuttonTest(AbstractLabelTest, unittest.TestCase): - OPTIONS = ( - 'class', 'command', 'compound', 'cursor', - 'image', - 'offvalue', 'onvalue', - 'padding', 'state', 'style', - 'takefocus', 'text', 'textvariable', - 'underline', 'variable', 'width', - ) - - def create(self, **kwargs): - return ttk.Checkbutton(self.root, **kwargs) - - def test_offvalue(self): - widget = self.create() - self.checkParams(widget, 'offvalue', 1, 2.3, '', 'any string') - - def test_onvalue(self): - widget = self.create() - self.checkParams(widget, 'onvalue', 1, 2.3, '', 'any string') - - def test_invoke(self): - success = [] - def cb_test(): - success.append(1) - return "cb test called" - - cbtn = ttk.Checkbutton(self.root, command=cb_test) - # the variable automatically created by ttk.Checkbutton is actually - # undefined till we invoke the Checkbutton - self.assertEqual(cbtn.state(), ('alternate', )) - self.assertRaises(tkinter.TclError, cbtn.tk.globalgetvar, - cbtn['variable']) - - res = cbtn.invoke() - self.assertEqual(res, "cb test called") - self.assertEqual(cbtn['onvalue'], - cbtn.tk.globalgetvar(cbtn['variable'])) - self.assertTrue(success) - - cbtn['command'] = '' - res = cbtn.invoke() - self.assertFalse(str(res)) - self.assertLessEqual(len(success), 1) - self.assertEqual(cbtn['offvalue'], - cbtn.tk.globalgetvar(cbtn['variable'])) - - -@add_standard_options(IntegerSizeTests, StandardTtkOptionsTests) -class EntryTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'background', 'class', 'cursor', - 'exportselection', 'font', 'foreground', - 'invalidcommand', 'justify', - 'show', 'state', 'style', 'takefocus', 'textvariable', - 'validate', 'validatecommand', 'width', 'xscrollcommand', - ) - - def setUp(self): - super().setUp() - self.entry = self.create() - - def create(self, **kwargs): - return ttk.Entry(self.root, **kwargs) - - def test_invalidcommand(self): - widget = self.create() - self.checkCommandParam(widget, 'invalidcommand') - - def test_show(self): - widget = self.create() - self.checkParam(widget, 'show', '*') - self.checkParam(widget, 'show', '') - self.checkParam(widget, 'show', ' ') - - def test_state(self): - widget = self.create() - self.checkParams(widget, 'state', - 'disabled', 'normal', 'readonly') - - def test_validate(self): - widget = self.create() - self.checkEnumParam(widget, 'validate', - 'all', 'key', 'focus', 'focusin', 'focusout', 'none') - - def test_validatecommand(self): - widget = self.create() - self.checkCommandParam(widget, 'validatecommand') - - - def test_bbox(self): - self.assertIsBoundingBox(self.entry.bbox(0)) - self.assertRaises(tkinter.TclError, self.entry.bbox, 'noindex') - self.assertRaises(tkinter.TclError, self.entry.bbox, None) - - - def test_identify(self): - self.entry.pack() - self.entry.wait_visibility() - self.entry.update_idletasks() - - self.assertEqual(self.entry.identify(5, 5), "textarea") - self.assertEqual(self.entry.identify(-1, -1), "") - - self.assertRaises(tkinter.TclError, self.entry.identify, None, 5) - self.assertRaises(tkinter.TclError, self.entry.identify, 5, None) - self.assertRaises(tkinter.TclError, self.entry.identify, 5, '') - - - def test_validation_options(self): - success = [] - test_invalid = lambda: success.append(True) - - self.entry['validate'] = 'none' - self.entry['validatecommand'] = lambda: False - - self.entry['invalidcommand'] = test_invalid - self.entry.validate() - self.assertTrue(success) - - self.entry['invalidcommand'] = '' - self.entry.validate() - self.assertEqual(len(success), 1) - - self.entry['invalidcommand'] = test_invalid - self.entry['validatecommand'] = lambda: True - self.entry.validate() - self.assertEqual(len(success), 1) - - self.entry['validatecommand'] = '' - self.entry.validate() - self.assertEqual(len(success), 1) - - self.entry['validatecommand'] = True - self.assertRaises(tkinter.TclError, self.entry.validate) - - - def test_validation(self): - validation = [] - def validate(to_insert): - if not 'a' <= to_insert.lower() <= 'z': - validation.append(False) - return False - validation.append(True) - return True - - self.entry['validate'] = 'key' - self.entry['validatecommand'] = self.entry.register(validate), '%S' - - self.entry.insert('end', 1) - self.entry.insert('end', 'a') - self.assertEqual(validation, [False, True]) - self.assertEqual(self.entry.get(), 'a') - - - def test_revalidation(self): - def validate(content): - for letter in content: - if not 'a' <= letter.lower() <= 'z': - return False - return True - - self.entry['validatecommand'] = self.entry.register(validate), '%P' - - self.entry.insert('end', 'avocado') - self.assertEqual(self.entry.validate(), True) - self.assertEqual(self.entry.state(), ()) - - self.entry.delete(0, 'end') - self.assertEqual(self.entry.get(), '') - - self.entry.insert('end', 'a1b') - self.assertEqual(self.entry.validate(), False) - self.assertEqual(self.entry.state(), ('invalid', )) - - self.entry.delete(1) - self.assertEqual(self.entry.validate(), True) - self.assertEqual(self.entry.state(), ()) - - -@add_standard_options(IntegerSizeTests, StandardTtkOptionsTests) -class ComboboxTest(EntryTest, unittest.TestCase): - OPTIONS = ( - 'background', 'class', 'cursor', 'exportselection', - 'font', 'foreground', 'height', 'invalidcommand', - 'justify', 'postcommand', 'show', 'state', 'style', - 'takefocus', 'textvariable', - 'validate', 'validatecommand', 'values', - 'width', 'xscrollcommand', - ) - - def setUp(self): - super().setUp() - self.combo = self.create() - - def create(self, **kwargs): - return ttk.Combobox(self.root, **kwargs) - - def test_height(self): - widget = self.create() - self.checkParams(widget, 'height', 100, 101.2, 102.6, -100, 0, '1i') - - def _show_drop_down_listbox(self): - width = self.combo.winfo_width() - self.combo.event_generate('<ButtonPress-1>', x=width - 5, y=5) - self.combo.event_generate('<ButtonRelease-1>', x=width - 5, y=5) - self.combo.update_idletasks() - - - def test_virtual_event(self): - success = [] - - self.combo['values'] = [1] - self.combo.bind('<<ComboboxSelected>>', - lambda evt: success.append(True)) - self.combo.pack() - self.combo.wait_visibility() - - height = self.combo.winfo_height() - self._show_drop_down_listbox() - self.combo.update() - self.combo.event_generate('<Return>') - self.combo.update() - - self.assertTrue(success) - - - def test_postcommand(self): - success = [] - - self.combo['postcommand'] = lambda: success.append(True) - self.combo.pack() - self.combo.wait_visibility() - - self._show_drop_down_listbox() - self.assertTrue(success) - - # testing postcommand removal - self.combo['postcommand'] = '' - self._show_drop_down_listbox() - self.assertEqual(len(success), 1) - - - def test_values(self): - def check_get_current(getval, currval): - self.assertEqual(self.combo.get(), getval) - self.assertEqual(self.combo.current(), currval) - - self.assertEqual(self.combo['values'], - () if tcl_version < (8, 5) else '') - check_get_current('', -1) - - self.checkParam(self.combo, 'values', 'mon tue wed thur', - expected=('mon', 'tue', 'wed', 'thur')) - self.checkParam(self.combo, 'values', ('mon', 'tue', 'wed', 'thur')) - self.checkParam(self.combo, 'values', (42, 3.14, '', 'any string')) - self.checkParam(self.combo, 'values', '', - expected='' if get_tk_patchlevel() < (8, 5, 10) else ()) - - self.combo['values'] = ['a', 1, 'c'] - - self.combo.set('c') - check_get_current('c', 2) - - self.combo.current(0) - check_get_current('a', 0) - - self.combo.set('d') - check_get_current('d', -1) - - # testing values with empty string - self.combo.set('') - self.combo['values'] = (1, 2, '', 3) - check_get_current('', 2) - - # testing values with empty string set through configure - self.combo.configure(values=[1, '', 2]) - self.assertEqual(self.combo['values'], - ('1', '', '2') if self.wantobjects else - '1 {} 2') - - # testing values with spaces - self.combo['values'] = ['a b', 'a\tb', 'a\nb'] - self.assertEqual(self.combo['values'], - ('a b', 'a\tb', 'a\nb') if self.wantobjects else - '{a b} {a\tb} {a\nb}') - - # testing values with special characters - self.combo['values'] = [r'a\tb', '"a"', '} {'] - self.assertEqual(self.combo['values'], - (r'a\tb', '"a"', '} {') if self.wantobjects else - r'a\\tb {"a"} \}\ \{') - - # out of range - self.assertRaises(tkinter.TclError, self.combo.current, - len(self.combo['values'])) - # it expects an integer (or something that can be converted to int) - self.assertRaises(tkinter.TclError, self.combo.current, '') - - # testing creating combobox with empty string in values - combo2 = ttk.Combobox(self.root, values=[1, 2, '']) - self.assertEqual(combo2['values'], - ('1', '2', '') if self.wantobjects else '1 2 {}') - combo2.destroy() - - -@add_standard_options(IntegerSizeTests, StandardTtkOptionsTests) -class PanedWindowTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'cursor', 'height', - 'orient', 'style', 'takefocus', 'width', - ) - - def setUp(self): - super().setUp() - self.paned = self.create() - - def create(self, **kwargs): - return ttk.PanedWindow(self.root, **kwargs) - - def test_orient(self): - widget = self.create() - self.assertEqual(str(widget['orient']), 'vertical') - errmsg='attempt to change read-only option' - if get_tk_patchlevel() < (8, 6, 0, 'beta', 3): - errmsg='Attempt to change read-only option' - self.checkInvalidParam(widget, 'orient', 'horizontal', - errmsg=errmsg) - widget2 = self.create(orient='horizontal') - self.assertEqual(str(widget2['orient']), 'horizontal') - - def test_add(self): - # attempt to add a child that is not a direct child of the paned window - label = ttk.Label(self.paned) - child = ttk.Label(label) - self.assertRaises(tkinter.TclError, self.paned.add, child) - label.destroy() - child.destroy() - # another attempt - label = ttk.Label(self.root) - child = ttk.Label(label) - self.assertRaises(tkinter.TclError, self.paned.add, child) - child.destroy() - label.destroy() - - good_child = ttk.Label(self.root) - self.paned.add(good_child) - # re-adding a child is not accepted - self.assertRaises(tkinter.TclError, self.paned.add, good_child) - - other_child = ttk.Label(self.paned) - self.paned.add(other_child) - self.assertEqual(self.paned.pane(0), self.paned.pane(1)) - self.assertRaises(tkinter.TclError, self.paned.pane, 2) - good_child.destroy() - other_child.destroy() - self.assertRaises(tkinter.TclError, self.paned.pane, 0) - - - def test_forget(self): - self.assertRaises(tkinter.TclError, self.paned.forget, None) - self.assertRaises(tkinter.TclError, self.paned.forget, 0) - - self.paned.add(ttk.Label(self.root)) - self.paned.forget(0) - self.assertRaises(tkinter.TclError, self.paned.forget, 0) - - - def test_insert(self): - self.assertRaises(tkinter.TclError, self.paned.insert, None, 0) - self.assertRaises(tkinter.TclError, self.paned.insert, 0, None) - self.assertRaises(tkinter.TclError, self.paned.insert, 0, 0) - - child = ttk.Label(self.root) - child2 = ttk.Label(self.root) - child3 = ttk.Label(self.root) - - self.assertRaises(tkinter.TclError, self.paned.insert, 0, child) - - self.paned.insert('end', child2) - self.paned.insert(0, child) - self.assertEqual(self.paned.panes(), (str(child), str(child2))) - - self.paned.insert(0, child2) - self.assertEqual(self.paned.panes(), (str(child2), str(child))) - - self.paned.insert('end', child3) - self.assertEqual(self.paned.panes(), - (str(child2), str(child), str(child3))) - - # reinserting a child should move it to its current position - panes = self.paned.panes() - self.paned.insert('end', child3) - self.assertEqual(panes, self.paned.panes()) - - # moving child3 to child2 position should result in child2 ending up - # in previous child position and child ending up in previous child3 - # position - self.paned.insert(child2, child3) - self.assertEqual(self.paned.panes(), - (str(child3), str(child2), str(child))) - - - def test_pane(self): - self.assertRaises(tkinter.TclError, self.paned.pane, 0) - - child = ttk.Label(self.root) - self.paned.add(child) - self.assertIsInstance(self.paned.pane(0), dict) - self.assertEqual(self.paned.pane(0, weight=None), - 0 if self.wantobjects else '0') - # newer form for querying a single option - self.assertEqual(self.paned.pane(0, 'weight'), - 0 if self.wantobjects else '0') - self.assertEqual(self.paned.pane(0), self.paned.pane(str(child))) - - self.assertRaises(tkinter.TclError, self.paned.pane, 0, - badoption='somevalue') - - - def test_sashpos(self): - self.assertRaises(tkinter.TclError, self.paned.sashpos, None) - self.assertRaises(tkinter.TclError, self.paned.sashpos, '') - self.assertRaises(tkinter.TclError, self.paned.sashpos, 0) - - child = ttk.Label(self.paned, text='a') - self.paned.add(child, weight=1) - self.assertRaises(tkinter.TclError, self.paned.sashpos, 0) - child2 = ttk.Label(self.paned, text='b') - self.paned.add(child2) - self.assertRaises(tkinter.TclError, self.paned.sashpos, 1) - - self.paned.pack(expand=True, fill='both') - self.paned.wait_visibility() - - curr_pos = self.paned.sashpos(0) - self.paned.sashpos(0, 1000) - self.assertNotEqual(curr_pos, self.paned.sashpos(0)) - self.assertIsInstance(self.paned.sashpos(0), int) - - -@add_standard_options(StandardTtkOptionsTests) -class RadiobuttonTest(AbstractLabelTest, unittest.TestCase): - OPTIONS = ( - 'class', 'command', 'compound', 'cursor', - 'image', - 'padding', 'state', 'style', - 'takefocus', 'text', 'textvariable', - 'underline', 'value', 'variable', 'width', - ) - - def create(self, **kwargs): - return ttk.Radiobutton(self.root, **kwargs) - - def test_value(self): - widget = self.create() - self.checkParams(widget, 'value', 1, 2.3, '', 'any string') - - def test_invoke(self): - success = [] - def cb_test(): - success.append(1) - return "cb test called" - - myvar = tkinter.IntVar(self.root) - cbtn = ttk.Radiobutton(self.root, command=cb_test, - variable=myvar, value=0) - cbtn2 = ttk.Radiobutton(self.root, command=cb_test, - variable=myvar, value=1) - - if self.wantobjects: - conv = lambda x: x - else: - conv = int - - res = cbtn.invoke() - self.assertEqual(res, "cb test called") - self.assertEqual(conv(cbtn['value']), myvar.get()) - self.assertEqual(myvar.get(), - conv(cbtn.tk.globalgetvar(cbtn['variable']))) - self.assertTrue(success) - - cbtn2['command'] = '' - res = cbtn2.invoke() - self.assertEqual(str(res), '') - self.assertLessEqual(len(success), 1) - self.assertEqual(conv(cbtn2['value']), myvar.get()) - self.assertEqual(myvar.get(), - conv(cbtn.tk.globalgetvar(cbtn['variable']))) - - self.assertEqual(str(cbtn['variable']), str(cbtn2['variable'])) - - -class MenubuttonTest(AbstractLabelTest, unittest.TestCase): - OPTIONS = ( - 'class', 'compound', 'cursor', 'direction', - 'image', 'menu', 'padding', 'state', 'style', - 'takefocus', 'text', 'textvariable', - 'underline', 'width', - ) - - def create(self, **kwargs): - return ttk.Menubutton(self.root, **kwargs) - - def test_direction(self): - widget = self.create() - self.checkEnumParam(widget, 'direction', - 'above', 'below', 'left', 'right', 'flush') - - def test_menu(self): - widget = self.create() - menu = tkinter.Menu(widget, name='menu') - self.checkParam(widget, 'menu', menu, conv=str) - menu.destroy() - - -@add_standard_options(StandardTtkOptionsTests) -class ScaleTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'command', 'cursor', 'from', 'length', - 'orient', 'style', 'takefocus', 'to', 'value', 'variable', - ) - _conv_pixels = noconv - default_orient = 'horizontal' - - def setUp(self): - super().setUp() - self.scale = self.create() - self.scale.pack() - self.scale.update() - - def create(self, **kwargs): - return ttk.Scale(self.root, **kwargs) - - def test_from(self): - widget = self.create() - self.checkFloatParam(widget, 'from', 100, 14.9, 15.1, conv=False) - - def test_length(self): - widget = self.create() - self.checkPixelsParam(widget, 'length', 130, 131.2, 135.6, '5i') - - def test_to(self): - widget = self.create() - self.checkFloatParam(widget, 'to', 300, 14.9, 15.1, -10, conv=False) - - def test_value(self): - widget = self.create() - self.checkFloatParam(widget, 'value', 300, 14.9, 15.1, -10, conv=False) - - def test_custom_event(self): - failure = [1, 1, 1] # will need to be empty - - funcid = self.scale.bind('<<RangeChanged>>', lambda evt: failure.pop()) - - self.scale['from'] = 10 - self.scale['from_'] = 10 - self.scale['to'] = 3 - - self.assertFalse(failure) - - failure = [1, 1, 1] - self.scale.configure(from_=2, to=5) - self.scale.configure(from_=0, to=-2) - self.scale.configure(to=10) - - self.assertFalse(failure) - - - def test_get(self): - if self.wantobjects: - conv = lambda x: x - else: - conv = float - - scale_width = self.scale.winfo_width() - self.assertEqual(self.scale.get(scale_width, 0), self.scale['to']) - - self.assertEqual(conv(self.scale.get(0, 0)), conv(self.scale['from'])) - self.assertEqual(self.scale.get(), self.scale['value']) - self.scale['value'] = 30 - self.assertEqual(self.scale.get(), self.scale['value']) - - self.assertRaises(tkinter.TclError, self.scale.get, '', 0) - self.assertRaises(tkinter.TclError, self.scale.get, 0, '') - - - def test_set(self): - if self.wantobjects: - conv = lambda x: x - else: - conv = float - - # set restricts the max/min values according to the current range - max = conv(self.scale['to']) - new_max = max + 10 - self.scale.set(new_max) - self.assertEqual(conv(self.scale.get()), max) - min = conv(self.scale['from']) - self.scale.set(min - 1) - self.assertEqual(conv(self.scale.get()), min) - - # changing directly the variable doesn't impose this limitation tho - var = tkinter.DoubleVar(self.root) - self.scale['variable'] = var - var.set(max + 5) - self.assertEqual(conv(self.scale.get()), var.get()) - self.assertEqual(conv(self.scale.get()), max + 5) - del var - - # the same happens with the value option - self.scale['value'] = max + 10 - self.assertEqual(conv(self.scale.get()), max + 10) - self.assertEqual(conv(self.scale.get()), conv(self.scale['value'])) - - # nevertheless, note that the max/min values we can get specifying - # x, y coords are the ones according to the current range - self.assertEqual(conv(self.scale.get(0, 0)), min) - self.assertEqual(conv(self.scale.get(self.scale.winfo_width(), 0)), max) - - self.assertRaises(tkinter.TclError, self.scale.set, None) - - -@add_standard_options(StandardTtkOptionsTests) -class ProgressbarTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'cursor', 'orient', 'length', - 'mode', 'maximum', 'phase', - 'style', 'takefocus', 'value', 'variable', - ) - _conv_pixels = noconv - default_orient = 'horizontal' - - def create(self, **kwargs): - return ttk.Progressbar(self.root, **kwargs) - - def test_length(self): - widget = self.create() - self.checkPixelsParam(widget, 'length', 100.1, 56.7, '2i') - - def test_maximum(self): - widget = self.create() - self.checkFloatParam(widget, 'maximum', 150.2, 77.7, 0, -10, conv=False) - - def test_mode(self): - widget = self.create() - self.checkEnumParam(widget, 'mode', 'determinate', 'indeterminate') - - def test_phase(self): - # XXX - pass - - def test_value(self): - widget = self.create() - self.checkFloatParam(widget, 'value', 150.2, 77.7, 0, -10, - conv=False) - - -@unittest.skipIf(sys.platform == 'darwin', - 'ttk.Scrollbar is special on MacOSX') -@add_standard_options(StandardTtkOptionsTests) -class ScrollbarTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'command', 'cursor', 'orient', 'style', 'takefocus', - ) - default_orient = 'vertical' - - def create(self, **kwargs): - return ttk.Scrollbar(self.root, **kwargs) - - -@add_standard_options(IntegerSizeTests, StandardTtkOptionsTests) -class NotebookTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'cursor', 'height', 'padding', 'style', 'takefocus', 'width', - ) - - def setUp(self): - super().setUp() - self.nb = self.create(padding=0) - self.child1 = ttk.Label(self.root) - self.child2 = ttk.Label(self.root) - self.nb.add(self.child1, text='a') - self.nb.add(self.child2, text='b') - - def create(self, **kwargs): - return ttk.Notebook(self.root, **kwargs) - - def test_tab_identifiers(self): - self.nb.forget(0) - self.nb.hide(self.child2) - self.assertRaises(tkinter.TclError, self.nb.tab, self.child1) - self.assertEqual(self.nb.index('end'), 1) - self.nb.add(self.child2) - self.assertEqual(self.nb.index('end'), 1) - self.nb.select(self.child2) - - self.assertTrue(self.nb.tab('current')) - self.nb.add(self.child1, text='a') - - self.nb.pack() - self.nb.wait_visibility() - if sys.platform == 'darwin': - tb_idx = "@20,5" - else: - tb_idx = "@5,5" - self.assertEqual(self.nb.tab(tb_idx), self.nb.tab('current')) - - for i in range(5, 100, 5): - try: - if self.nb.tab('@%d, 5' % i, text=None) == 'a': - break - except tkinter.TclError: - pass - - else: - self.fail("Tab with text 'a' not found") - - - def test_add_and_hidden(self): - self.assertRaises(tkinter.TclError, self.nb.hide, -1) - self.assertRaises(tkinter.TclError, self.nb.hide, 'hi') - self.assertRaises(tkinter.TclError, self.nb.hide, None) - self.assertRaises(tkinter.TclError, self.nb.add, None) - self.assertRaises(tkinter.TclError, self.nb.add, ttk.Label(self.root), - unknown='option') - - tabs = self.nb.tabs() - self.nb.hide(self.child1) - self.nb.add(self.child1) - self.assertEqual(self.nb.tabs(), tabs) - - child = ttk.Label(self.root) - self.nb.add(child, text='c') - tabs = self.nb.tabs() - - curr = self.nb.index('current') - # verify that the tab gets readded at its previous position - child2_index = self.nb.index(self.child2) - self.nb.hide(self.child2) - self.nb.add(self.child2) - self.assertEqual(self.nb.tabs(), tabs) - self.assertEqual(self.nb.index(self.child2), child2_index) - self.assertEqual(str(self.child2), self.nb.tabs()[child2_index]) - # but the tab next to it (not hidden) is the one selected now - self.assertEqual(self.nb.index('current'), curr + 1) - - - def test_forget(self): - self.assertRaises(tkinter.TclError, self.nb.forget, -1) - self.assertRaises(tkinter.TclError, self.nb.forget, 'hi') - self.assertRaises(tkinter.TclError, self.nb.forget, None) - - tabs = self.nb.tabs() - child1_index = self.nb.index(self.child1) - self.nb.forget(self.child1) - self.assertNotIn(str(self.child1), self.nb.tabs()) - self.assertEqual(len(tabs) - 1, len(self.nb.tabs())) - - self.nb.add(self.child1) - self.assertEqual(self.nb.index(self.child1), 1) - self.assertNotEqual(child1_index, self.nb.index(self.child1)) - - - def test_index(self): - self.assertRaises(tkinter.TclError, self.nb.index, -1) - self.assertRaises(tkinter.TclError, self.nb.index, None) - - self.assertIsInstance(self.nb.index('end'), int) - self.assertEqual(self.nb.index(self.child1), 0) - self.assertEqual(self.nb.index(self.child2), 1) - self.assertEqual(self.nb.index('end'), 2) - - - def test_insert(self): - # moving tabs - tabs = self.nb.tabs() - self.nb.insert(1, tabs[0]) - self.assertEqual(self.nb.tabs(), (tabs[1], tabs[0])) - self.nb.insert(self.child1, self.child2) - self.assertEqual(self.nb.tabs(), tabs) - self.nb.insert('end', self.child1) - self.assertEqual(self.nb.tabs(), (tabs[1], tabs[0])) - self.nb.insert('end', 0) - self.assertEqual(self.nb.tabs(), tabs) - # bad moves - self.assertRaises(tkinter.TclError, self.nb.insert, 2, tabs[0]) - self.assertRaises(tkinter.TclError, self.nb.insert, -1, tabs[0]) - - # new tab - child3 = ttk.Label(self.root) - self.nb.insert(1, child3) - self.assertEqual(self.nb.tabs(), (tabs[0], str(child3), tabs[1])) - self.nb.forget(child3) - self.assertEqual(self.nb.tabs(), tabs) - self.nb.insert(self.child1, child3) - self.assertEqual(self.nb.tabs(), (str(child3), ) + tabs) - self.nb.forget(child3) - self.assertRaises(tkinter.TclError, self.nb.insert, 2, child3) - self.assertRaises(tkinter.TclError, self.nb.insert, -1, child3) - - # bad inserts - self.assertRaises(tkinter.TclError, self.nb.insert, 'end', None) - self.assertRaises(tkinter.TclError, self.nb.insert, None, 0) - self.assertRaises(tkinter.TclError, self.nb.insert, None, None) - - - def test_select(self): - self.nb.pack() - self.nb.wait_visibility() - - success = [] - tab_changed = [] - - self.child1.bind('<Unmap>', lambda evt: success.append(True)) - self.nb.bind('<<NotebookTabChanged>>', - lambda evt: tab_changed.append(True)) - - self.assertEqual(self.nb.select(), str(self.child1)) - self.nb.select(self.child2) - self.assertTrue(success) - self.assertEqual(self.nb.select(), str(self.child2)) - - self.nb.update() - self.assertTrue(tab_changed) - - - def test_tab(self): - self.assertRaises(tkinter.TclError, self.nb.tab, -1) - self.assertRaises(tkinter.TclError, self.nb.tab, 'notab') - self.assertRaises(tkinter.TclError, self.nb.tab, None) - - self.assertIsInstance(self.nb.tab(self.child1), dict) - self.assertEqual(self.nb.tab(self.child1, text=None), 'a') - # newer form for querying a single option - self.assertEqual(self.nb.tab(self.child1, 'text'), 'a') - self.nb.tab(self.child1, text='abc') - self.assertEqual(self.nb.tab(self.child1, text=None), 'abc') - self.assertEqual(self.nb.tab(self.child1, 'text'), 'abc') - - - def test_tabs(self): - self.assertEqual(len(self.nb.tabs()), 2) - - self.nb.forget(self.child1) - self.nb.forget(self.child2) - - self.assertEqual(self.nb.tabs(), ()) - - - def test_traversal(self): - self.nb.pack() - self.nb.wait_visibility() - - self.nb.select(0) - - simulate_mouse_click(self.nb, 5, 5) - self.nb.focus_force() - self.nb.event_generate('<Control-Tab>') - self.assertEqual(self.nb.select(), str(self.child2)) - self.nb.focus_force() - self.nb.event_generate('<Shift-Control-Tab>') - self.assertEqual(self.nb.select(), str(self.child1)) - self.nb.focus_force() - self.nb.event_generate('<Shift-Control-Tab>') - self.assertEqual(self.nb.select(), str(self.child2)) - - self.nb.tab(self.child1, text='a', underline=0) - self.nb.enable_traversal() - self.nb.focus_force() - simulate_mouse_click(self.nb, 5, 5) - if sys.platform == 'darwin': - self.nb.event_generate('<Option-a>') - else: - self.nb.event_generate('<Alt-a>') - self.assertEqual(self.nb.select(), str(self.child1)) - -@add_standard_options(IntegerSizeTests, StandardTtkOptionsTests) -class SpinboxTest(EntryTest, unittest.TestCase): - OPTIONS = ( - 'background', 'class', 'command', 'cursor', 'exportselection', - 'font', 'foreground', 'format', 'from', 'increment', - 'invalidcommand', 'justify', 'show', 'state', 'style', - 'takefocus', 'textvariable', 'to', 'validate', 'validatecommand', - 'values', 'width', 'wrap', 'xscrollcommand', - ) - - def setUp(self): - super().setUp() - self.spin = self.create() - self.spin.pack() - - def create(self, **kwargs): - return ttk.Spinbox(self.root, **kwargs) - - def _click_increment_arrow(self): - width = self.spin.winfo_width() - height = self.spin.winfo_height() - x = width - 5 - y = height//2 - 5 - self.spin.event_generate('<ButtonPress-1>', x=x, y=y) - self.spin.event_generate('<ButtonRelease-1>', x=x, y=y) - self.spin.update_idletasks() - - def _click_decrement_arrow(self): - width = self.spin.winfo_width() - height = self.spin.winfo_height() - x = width - 5 - y = height//2 + 4 - self.spin.event_generate('<ButtonPress-1>', x=x, y=y) - self.spin.event_generate('<ButtonRelease-1>', x=x, y=y) - self.spin.update_idletasks() - - def test_command(self): - success = [] - - self.spin['command'] = lambda: success.append(True) - self.spin.update() - self._click_increment_arrow() - self.spin.update() - self.assertTrue(success) - - self._click_decrement_arrow() - self.assertEqual(len(success), 2) - - # testing postcommand removal - self.spin['command'] = '' - self.spin.update_idletasks() - self._click_increment_arrow() - self._click_decrement_arrow() - self.spin.update() - self.assertEqual(len(success), 2) - - def test_to(self): - self.spin['from'] = 0 - self.spin['to'] = 5 - self.spin.set(4) - self.spin.update() - self._click_increment_arrow() # 5 - - self.assertEqual(self.spin.get(), '5') - - self._click_increment_arrow() # 5 - self.assertEqual(self.spin.get(), '5') - - def test_from(self): - self.spin['from'] = 1 - self.spin['to'] = 10 - self.spin.set(2) - self.spin.update() - self._click_decrement_arrow() # 1 - self.assertEqual(self.spin.get(), '1') - self._click_decrement_arrow() # 1 - self.assertEqual(self.spin.get(), '1') - - def test_increment(self): - self.spin['from'] = 0 - self.spin['to'] = 10 - self.spin['increment'] = 4 - self.spin.set(1) - self.spin.update() - - self._click_increment_arrow() # 5 - self.assertEqual(self.spin.get(), '5') - self.spin['increment'] = 2 - self.spin.update() - self._click_decrement_arrow() # 3 - self.assertEqual(self.spin.get(), '3') - - def test_format(self): - self.spin.set(1) - self.spin['format'] = '%10.3f' - self.spin.update() - self._click_increment_arrow() - value = self.spin.get() - - self.assertEqual(len(value), 10) - self.assertEqual(value.index('.'), 6) - - self.spin['format'] = '' - self.spin.update() - self._click_increment_arrow() - value = self.spin.get() - self.assertTrue('.' not in value) - self.assertEqual(len(value), 1) - - def test_wrap(self): - self.spin['to'] = 10 - self.spin['from'] = 1 - self.spin.set(1) - self.spin['wrap'] = True - self.spin.update() - - self._click_decrement_arrow() - self.assertEqual(self.spin.get(), '10') - - self._click_increment_arrow() - self.assertEqual(self.spin.get(), '1') - - self.spin['wrap'] = False - self.spin.update() - - self._click_decrement_arrow() - self.assertEqual(self.spin.get(), '1') - - def test_values(self): - self.assertEqual(self.spin['values'], - () if tcl_version < (8, 5) else '') - self.checkParam(self.spin, 'values', 'mon tue wed thur', - expected=('mon', 'tue', 'wed', 'thur')) - self.checkParam(self.spin, 'values', ('mon', 'tue', 'wed', 'thur')) - self.checkParam(self.spin, 'values', (42, 3.14, '', 'any string')) - self.checkParam( - self.spin, - 'values', - '', - expected='' if get_tk_patchlevel() < (8, 5, 10) else () - ) - - self.spin['values'] = ['a', 1, 'c'] - - # test incrementing / decrementing values - self.spin.set('a') - self.spin.update() - self._click_increment_arrow() - self.assertEqual(self.spin.get(), '1') - - self._click_decrement_arrow() - self.assertEqual(self.spin.get(), 'a') - - # testing values with empty string set through configure - self.spin.configure(values=[1, '', 2]) - self.assertEqual(self.spin['values'], - ('1', '', '2') if self.wantobjects else - '1 {} 2') - - # testing values with spaces - self.spin['values'] = ['a b', 'a\tb', 'a\nb'] - self.assertEqual(self.spin['values'], - ('a b', 'a\tb', 'a\nb') if self.wantobjects else - '{a b} {a\tb} {a\nb}') - - # testing values with special characters - self.spin['values'] = [r'a\tb', '"a"', '} {'] - self.assertEqual(self.spin['values'], - (r'a\tb', '"a"', '} {') if self.wantobjects else - r'a\\tb {"a"} \}\ \{') - - # testing creating spinbox with empty string in values - spin2 = ttk.Spinbox(self.root, values=[1, 2, '']) - self.assertEqual(spin2['values'], - ('1', '2', '') if self.wantobjects else '1 2 {}') - spin2.destroy() - - -@add_standard_options(StandardTtkOptionsTests) -class TreeviewTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'columns', 'cursor', 'displaycolumns', - 'height', 'padding', 'selectmode', 'show', - 'style', 'takefocus', 'xscrollcommand', 'yscrollcommand', - ) - - def setUp(self): - super().setUp() - self.tv = self.create(padding=0) - - def create(self, **kwargs): - return ttk.Treeview(self.root, **kwargs) - - def test_columns(self): - widget = self.create() - self.checkParam(widget, 'columns', 'a b c', - expected=('a', 'b', 'c')) - self.checkParam(widget, 'columns', ('a', 'b', 'c')) - self.checkParam(widget, 'columns', (), - expected='' if get_tk_patchlevel() < (8, 5, 10) else ()) - - def test_displaycolumns(self): - widget = self.create() - widget['columns'] = ('a', 'b', 'c') - self.checkParam(widget, 'displaycolumns', 'b a c', - expected=('b', 'a', 'c')) - self.checkParam(widget, 'displaycolumns', ('b', 'a', 'c')) - self.checkParam(widget, 'displaycolumns', '#all', - expected=('#all',)) - self.checkParam(widget, 'displaycolumns', (2, 1, 0)) - self.checkInvalidParam(widget, 'displaycolumns', ('a', 'b', 'd'), - errmsg='Invalid column index d') - self.checkInvalidParam(widget, 'displaycolumns', (1, 2, 3), - errmsg='Column index 3 out of bounds') - self.checkInvalidParam(widget, 'displaycolumns', (1, -2), - errmsg='Column index -2 out of bounds') - - def test_height(self): - widget = self.create() - self.checkPixelsParam(widget, 'height', 100, -100, 0, '3c', conv=False) - self.checkPixelsParam(widget, 'height', 101.2, 102.6, conv=noconv) - - def test_selectmode(self): - widget = self.create() - self.checkEnumParam(widget, 'selectmode', - 'none', 'browse', 'extended') - - def test_show(self): - widget = self.create() - self.checkParam(widget, 'show', 'tree headings', - expected=('tree', 'headings')) - self.checkParam(widget, 'show', ('tree', 'headings')) - self.checkParam(widget, 'show', ('headings', 'tree')) - self.checkParam(widget, 'show', 'tree', expected=('tree',)) - self.checkParam(widget, 'show', 'headings', expected=('headings',)) - - def test_bbox(self): - self.tv.pack() - self.assertEqual(self.tv.bbox(''), '') - self.tv.wait_visibility() - self.tv.update() - - item_id = self.tv.insert('', 'end') - children = self.tv.get_children() - self.assertTrue(children) - - bbox = self.tv.bbox(children[0]) - self.assertIsBoundingBox(bbox) - - # compare width in bboxes - self.tv['columns'] = ['test'] - self.tv.column('test', width=50) - bbox_column0 = self.tv.bbox(children[0], 0) - root_width = self.tv.column('#0', width=None) - if not self.wantobjects: - root_width = int(root_width) - self.assertEqual(bbox_column0[0], bbox[0] + root_width) - - # verify that bbox of a closed item is the empty string - child1 = self.tv.insert(item_id, 'end') - self.assertEqual(self.tv.bbox(child1), '') - - - def test_children(self): - # no children yet, should get an empty tuple - self.assertEqual(self.tv.get_children(), ()) - - item_id = self.tv.insert('', 'end') - self.assertIsInstance(self.tv.get_children(), tuple) - self.assertEqual(self.tv.get_children()[0], item_id) - - # add item_id and child3 as children of child2 - child2 = self.tv.insert('', 'end') - child3 = self.tv.insert('', 'end') - self.tv.set_children(child2, item_id, child3) - self.assertEqual(self.tv.get_children(child2), (item_id, child3)) - - # child3 has child2 as parent, thus trying to set child2 as a children - # of child3 should result in an error - self.assertRaises(tkinter.TclError, - self.tv.set_children, child3, child2) - - # remove child2 children - self.tv.set_children(child2) - self.assertEqual(self.tv.get_children(child2), ()) - - # remove root's children - self.tv.set_children('') - self.assertEqual(self.tv.get_children(), ()) - - - def test_column(self): - # return a dict with all options/values - self.assertIsInstance(self.tv.column('#0'), dict) - # return a single value of the given option - if self.wantobjects: - self.assertIsInstance(self.tv.column('#0', width=None), int) - # set a new value for an option - self.tv.column('#0', width=10) - # testing new way to get option value - self.assertEqual(self.tv.column('#0', 'width'), - 10 if self.wantobjects else '10') - self.assertEqual(self.tv.column('#0', width=None), - 10 if self.wantobjects else '10') - # check read-only option - self.assertRaises(tkinter.TclError, self.tv.column, '#0', id='X') - - self.assertRaises(tkinter.TclError, self.tv.column, 'invalid') - invalid_kws = [ - {'unknown_option': 'some value'}, {'stretch': 'wrong'}, - {'anchor': 'wrong'}, {'width': 'wrong'}, {'minwidth': 'wrong'} - ] - for kw in invalid_kws: - self.assertRaises(tkinter.TclError, self.tv.column, '#0', - **kw) - - - def test_delete(self): - self.assertRaises(tkinter.TclError, self.tv.delete, '#0') - - item_id = self.tv.insert('', 'end') - item2 = self.tv.insert(item_id, 'end') - self.assertEqual(self.tv.get_children(), (item_id, )) - self.assertEqual(self.tv.get_children(item_id), (item2, )) - - self.tv.delete(item_id) - self.assertFalse(self.tv.get_children()) - - # reattach should fail - self.assertRaises(tkinter.TclError, - self.tv.reattach, item_id, '', 'end') - - # test multiple item delete - item1 = self.tv.insert('', 'end') - item2 = self.tv.insert('', 'end') - self.assertEqual(self.tv.get_children(), (item1, item2)) - - self.tv.delete(item1, item2) - self.assertFalse(self.tv.get_children()) - - - def test_detach_reattach(self): - item_id = self.tv.insert('', 'end') - item2 = self.tv.insert(item_id, 'end') - - # calling detach without items is valid, although it does nothing - prev = self.tv.get_children() - self.tv.detach() # this should do nothing - self.assertEqual(prev, self.tv.get_children()) - - self.assertEqual(self.tv.get_children(), (item_id, )) - self.assertEqual(self.tv.get_children(item_id), (item2, )) - - # detach item with children - self.tv.detach(item_id) - self.assertFalse(self.tv.get_children()) - - # reattach item with children - self.tv.reattach(item_id, '', 'end') - self.assertEqual(self.tv.get_children(), (item_id, )) - self.assertEqual(self.tv.get_children(item_id), (item2, )) - - # move a children to the root - self.tv.move(item2, '', 'end') - self.assertEqual(self.tv.get_children(), (item_id, item2)) - self.assertEqual(self.tv.get_children(item_id), ()) - - # bad values - self.assertRaises(tkinter.TclError, - self.tv.reattach, 'nonexistent', '', 'end') - self.assertRaises(tkinter.TclError, - self.tv.detach, 'nonexistent') - self.assertRaises(tkinter.TclError, - self.tv.reattach, item2, 'otherparent', 'end') - self.assertRaises(tkinter.TclError, - self.tv.reattach, item2, '', 'invalid') - - # multiple detach - self.tv.detach(item_id, item2) - self.assertEqual(self.tv.get_children(), ()) - self.assertEqual(self.tv.get_children(item_id), ()) - - - def test_exists(self): - self.assertEqual(self.tv.exists('something'), False) - self.assertEqual(self.tv.exists(''), True) - self.assertEqual(self.tv.exists({}), False) - - # the following will make a tk.call equivalent to - # tk.call(treeview, "exists") which should result in an error - # in the tcl interpreter since tk requires an item. - self.assertRaises(tkinter.TclError, self.tv.exists, None) - - - def test_focus(self): - # nothing is focused right now - self.assertEqual(self.tv.focus(), '') - - item1 = self.tv.insert('', 'end') - self.tv.focus(item1) - self.assertEqual(self.tv.focus(), item1) - - self.tv.delete(item1) - self.assertEqual(self.tv.focus(), '') - - # try focusing inexistent item - self.assertRaises(tkinter.TclError, self.tv.focus, 'hi') - - - def test_heading(self): - # check a dict is returned - self.assertIsInstance(self.tv.heading('#0'), dict) - - # check a value is returned - self.tv.heading('#0', text='hi') - self.assertEqual(self.tv.heading('#0', 'text'), 'hi') - self.assertEqual(self.tv.heading('#0', text=None), 'hi') - - # invalid option - self.assertRaises(tkinter.TclError, self.tv.heading, '#0', - background=None) - # invalid value - self.assertRaises(tkinter.TclError, self.tv.heading, '#0', - anchor=1) - - def test_heading_callback(self): - def simulate_heading_click(x, y): - simulate_mouse_click(self.tv, x, y) - self.tv.update() - - success = [] # no success for now - - self.tv.pack() - self.tv.wait_visibility() - self.tv.heading('#0', command=lambda: success.append(True)) - self.tv.column('#0', width=100) - self.tv.update() - - # assuming that the coords (5, 5) fall into heading #0 - simulate_heading_click(5, 5) - if not success: - self.fail("The command associated to the treeview heading wasn't " - "invoked.") - - success = [] - commands = self.tv.master._tclCommands - self.tv.heading('#0', command=str(self.tv.heading('#0', command=None))) - self.assertEqual(commands, self.tv.master._tclCommands) - simulate_heading_click(5, 5) - if not success: - self.fail("The command associated to the treeview heading wasn't " - "invoked.") - - # XXX The following raises an error in a tcl interpreter, but not in - # Python - #self.tv.heading('#0', command='I dont exist') - #simulate_heading_click(5, 5) - - - def test_index(self): - # item 'what' doesn't exist - self.assertRaises(tkinter.TclError, self.tv.index, 'what') - - self.assertEqual(self.tv.index(''), 0) - - item1 = self.tv.insert('', 'end') - item2 = self.tv.insert('', 'end') - c1 = self.tv.insert(item1, 'end') - c2 = self.tv.insert(item1, 'end') - self.assertEqual(self.tv.index(item1), 0) - self.assertEqual(self.tv.index(c1), 0) - self.assertEqual(self.tv.index(c2), 1) - self.assertEqual(self.tv.index(item2), 1) - - self.tv.move(item2, '', 0) - self.assertEqual(self.tv.index(item2), 0) - self.assertEqual(self.tv.index(item1), 1) - - # check that index still works even after its parent and siblings - # have been detached - self.tv.detach(item1) - self.assertEqual(self.tv.index(c2), 1) - self.tv.detach(c1) - self.assertEqual(self.tv.index(c2), 0) - - # but it fails after item has been deleted - self.tv.delete(item1) - self.assertRaises(tkinter.TclError, self.tv.index, c2) - - - def test_insert_item(self): - # parent 'none' doesn't exist - self.assertRaises(tkinter.TclError, self.tv.insert, 'none', 'end') - - # open values - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', - open='') - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', - open='please') - self.assertFalse(self.tv.delete(self.tv.insert('', 'end', open=True))) - self.assertFalse(self.tv.delete(self.tv.insert('', 'end', open=False))) - - # invalid index - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'middle') - - # trying to duplicate item id is invalid - itemid = self.tv.insert('', 'end', 'first-item') - self.assertEqual(itemid, 'first-item') - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', - 'first-item') - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', - MockTclObj('first-item')) - - # unicode values - value = '\xe1ba' - item = self.tv.insert('', 'end', values=(value, )) - self.assertEqual(self.tv.item(item, 'values'), - (value,) if self.wantobjects else value) - self.assertEqual(self.tv.item(item, values=None), - (value,) if self.wantobjects else value) - - self.tv.item(item, values=self.root.splitlist(self.tv.item(item, values=None))) - self.assertEqual(self.tv.item(item, values=None), - (value,) if self.wantobjects else value) - - self.assertIsInstance(self.tv.item(item), dict) - - # erase item values - self.tv.item(item, values='') - self.assertFalse(self.tv.item(item, values=None)) - - # item tags - item = self.tv.insert('', 'end', tags=[1, 2, value]) - self.assertEqual(self.tv.item(item, tags=None), - ('1', '2', value) if self.wantobjects else - '1 2 %s' % value) - self.tv.item(item, tags=[]) - self.assertFalse(self.tv.item(item, tags=None)) - self.tv.item(item, tags=(1, 2)) - self.assertEqual(self.tv.item(item, tags=None), - ('1', '2') if self.wantobjects else '1 2') - - # values with spaces - item = self.tv.insert('', 'end', values=('a b c', - '%s %s' % (value, value))) - self.assertEqual(self.tv.item(item, values=None), - ('a b c', '%s %s' % (value, value)) if self.wantobjects else - '{a b c} {%s %s}' % (value, value)) - - # text - self.assertEqual(self.tv.item( - self.tv.insert('', 'end', text="Label here"), text=None), - "Label here") - self.assertEqual(self.tv.item( - self.tv.insert('', 'end', text=value), text=None), - value) - - # test for values which are not None - itemid = self.tv.insert('', 'end', 0) - self.assertEqual(itemid, '0') - itemid = self.tv.insert('', 'end', 0.0) - self.assertEqual(itemid, '0.0') - # this is because False resolves to 0 and element with 0 iid is already present - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', False) - self.assertRaises(tkinter.TclError, self.tv.insert, '', 'end', '') - - - def test_selection(self): - self.assertRaises(TypeError, self.tv.selection, 'spam') - # item 'none' doesn't exist - self.assertRaises(tkinter.TclError, self.tv.selection_set, 'none') - self.assertRaises(tkinter.TclError, self.tv.selection_add, 'none') - self.assertRaises(tkinter.TclError, self.tv.selection_remove, 'none') - self.assertRaises(tkinter.TclError, self.tv.selection_toggle, 'none') - - item1 = self.tv.insert('', 'end') - item2 = self.tv.insert('', 'end') - c1 = self.tv.insert(item1, 'end') - c2 = self.tv.insert(item1, 'end') - c3 = self.tv.insert(item1, 'end') - self.assertEqual(self.tv.selection(), ()) - - self.tv.selection_set(c1, item2) - self.assertEqual(self.tv.selection(), (c1, item2)) - self.tv.selection_set(c2) - self.assertEqual(self.tv.selection(), (c2,)) - - self.tv.selection_add(c1, item2) - self.assertEqual(self.tv.selection(), (c1, c2, item2)) - self.tv.selection_add(item1) - self.assertEqual(self.tv.selection(), (item1, c1, c2, item2)) - self.tv.selection_add() - self.assertEqual(self.tv.selection(), (item1, c1, c2, item2)) - - self.tv.selection_remove(item1, c3) - self.assertEqual(self.tv.selection(), (c1, c2, item2)) - self.tv.selection_remove(c2) - self.assertEqual(self.tv.selection(), (c1, item2)) - self.tv.selection_remove() - self.assertEqual(self.tv.selection(), (c1, item2)) - - self.tv.selection_toggle(c1, c3) - self.assertEqual(self.tv.selection(), (c3, item2)) - self.tv.selection_toggle(item2) - self.assertEqual(self.tv.selection(), (c3,)) - self.tv.selection_toggle() - self.assertEqual(self.tv.selection(), (c3,)) - - self.tv.insert('', 'end', id='with spaces') - self.tv.selection_set('with spaces') - self.assertEqual(self.tv.selection(), ('with spaces',)) - - self.tv.insert('', 'end', id='{brace') - self.tv.selection_set('{brace') - self.assertEqual(self.tv.selection(), ('{brace',)) - - self.tv.insert('', 'end', id='unicode\u20ac') - self.tv.selection_set('unicode\u20ac') - self.assertEqual(self.tv.selection(), ('unicode\u20ac',)) - - self.tv.insert('', 'end', id=b'bytes\xe2\x82\xac') - self.tv.selection_set(b'bytes\xe2\x82\xac') - self.assertEqual(self.tv.selection(), ('bytes\xe2\x82\xac',)) - - self.tv.selection_set() - self.assertEqual(self.tv.selection(), ()) - - # Old interface - self.tv.selection_set((c1, item2)) - self.assertEqual(self.tv.selection(), (c1, item2)) - self.tv.selection_add((c1, item1)) - self.assertEqual(self.tv.selection(), (item1, c1, item2)) - self.tv.selection_remove((item1, c3)) - self.assertEqual(self.tv.selection(), (c1, item2)) - self.tv.selection_toggle((c1, c3)) - self.assertEqual(self.tv.selection(), (c3, item2)) - - - def test_set(self): - self.tv['columns'] = ['A', 'B'] - item = self.tv.insert('', 'end', values=['a', 'b']) - self.assertEqual(self.tv.set(item), {'A': 'a', 'B': 'b'}) - - self.tv.set(item, 'B', 'a') - self.assertEqual(self.tv.item(item, values=None), - ('a', 'a') if self.wantobjects else 'a a') - - self.tv['columns'] = ['B'] - self.assertEqual(self.tv.set(item), {'B': 'a'}) - - self.tv.set(item, 'B', 'b') - self.assertEqual(self.tv.set(item, column='B'), 'b') - self.assertEqual(self.tv.item(item, values=None), - ('b', 'a') if self.wantobjects else 'b a') - - self.tv.set(item, 'B', 123) - self.assertEqual(self.tv.set(item, 'B'), - 123 if self.wantobjects else '123') - self.assertEqual(self.tv.item(item, values=None), - (123, 'a') if self.wantobjects else '123 a') - self.assertEqual(self.tv.set(item), - {'B': 123} if self.wantobjects else {'B': '123'}) - - # inexistent column - self.assertRaises(tkinter.TclError, self.tv.set, item, 'A') - self.assertRaises(tkinter.TclError, self.tv.set, item, 'A', 'b') - - # inexistent item - self.assertRaises(tkinter.TclError, self.tv.set, 'notme') - - - def test_tag_bind(self): - events = [] - item1 = self.tv.insert('', 'end', tags=['call']) - item2 = self.tv.insert('', 'end', tags=['call']) - self.tv.tag_bind('call', '<ButtonPress-1>', - lambda evt: events.append(1)) - self.tv.tag_bind('call', '<ButtonRelease-1>', - lambda evt: events.append(2)) - - self.tv.pack() - self.tv.wait_visibility() - self.tv.update() - - pos_y = set() - found = set() - for i in range(0, 100, 10): - if len(found) == 2: # item1 and item2 already found - break - item_id = self.tv.identify_row(i) - if item_id and item_id not in found: - pos_y.add(i) - found.add(item_id) - - self.assertEqual(len(pos_y), 2) # item1 and item2 y pos - for y in pos_y: - simulate_mouse_click(self.tv, 0, y) - - # by now there should be 4 things in the events list, since each - # item had a bind for two events that were simulated above - self.assertEqual(len(events), 4) - for evt in zip(events[::2], events[1::2]): - self.assertEqual(evt, (1, 2)) - - - def test_tag_configure(self): - # Just testing parameter passing for now - self.assertRaises(TypeError, self.tv.tag_configure) - self.assertRaises(tkinter.TclError, self.tv.tag_configure, - 'test', sky='blue') - self.tv.tag_configure('test', foreground='blue') - self.assertEqual(str(self.tv.tag_configure('test', 'foreground')), - 'blue') - self.assertEqual(str(self.tv.tag_configure('test', foreground=None)), - 'blue') - self.assertIsInstance(self.tv.tag_configure('test'), dict) - - def test_tag_has(self): - item1 = self.tv.insert('', 'end', text='Item 1', tags=['tag1']) - item2 = self.tv.insert('', 'end', text='Item 2', tags=['tag2']) - self.assertRaises(TypeError, self.tv.tag_has) - self.assertRaises(TclError, self.tv.tag_has, 'tag1', 'non-existing') - self.assertTrue(self.tv.tag_has('tag1', item1)) - self.assertFalse(self.tv.tag_has('tag1', item2)) - self.assertFalse(self.tv.tag_has('tag2', item1)) - self.assertTrue(self.tv.tag_has('tag2', item2)) - self.assertFalse(self.tv.tag_has('tag3', item1)) - self.assertFalse(self.tv.tag_has('tag3', item2)) - self.assertEqual(self.tv.tag_has('tag1'), (item1,)) - self.assertEqual(self.tv.tag_has('tag2'), (item2,)) - self.assertEqual(self.tv.tag_has('tag3'), ()) - - -@add_standard_options(StandardTtkOptionsTests) -class SeparatorTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'cursor', 'orient', 'style', 'takefocus', - # 'state'? - ) - default_orient = 'horizontal' - - def create(self, **kwargs): - return ttk.Separator(self.root, **kwargs) - - -@add_standard_options(StandardTtkOptionsTests) -class SizegripTest(AbstractWidgetTest, unittest.TestCase): - OPTIONS = ( - 'class', 'cursor', 'style', 'takefocus', - # 'state'? - ) - - def create(self, **kwargs): - return ttk.Sizegrip(self.root, **kwargs) - -tests_gui = ( - ButtonTest, CheckbuttonTest, ComboboxTest, EntryTest, - FrameTest, LabelFrameTest, LabelTest, MenubuttonTest, - NotebookTest, PanedWindowTest, ProgressbarTest, - RadiobuttonTest, ScaleTest, ScrollbarTest, SeparatorTest, - SizegripTest, SpinboxTest, TreeviewTest, WidgetTest, - ) - -if __name__ == "__main__": - unittest.main() diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py deleted file mode 100644 index 573544d..0000000 --- a/Lib/tkinter/ttk.py +++ /dev/null @@ -1,1660 +0,0 @@ -"""Ttk wrapper. - -This module provides classes to allow using Tk themed widget set. - -Ttk is based on a revised and enhanced version of -TIP #48 (http://tip.tcl.tk/48) specified style engine. - -Its basic idea is to separate, to the extent possible, the code -implementing a widget's behavior from the code implementing its -appearance. Widget class bindings are primarily responsible for -maintaining the widget state and invoking callbacks, all aspects -of the widgets appearance lies at Themes. -""" - -__version__ = "0.3.1" - -__author__ = "Guilherme Polo <ggpolo@gmail.com>" - -__all__ = ["Button", "Checkbutton", "Combobox", "Entry", "Frame", "Label", - "Labelframe", "LabelFrame", "Menubutton", "Notebook", "Panedwindow", - "PanedWindow", "Progressbar", "Radiobutton", "Scale", "Scrollbar", - "Separator", "Sizegrip", "Spinbox", "Style", "Treeview", - # Extensions - "LabeledScale", "OptionMenu", - # functions - "tclobjs_to_py", "setup_master"] - -import tkinter -from tkinter import _flatten, _join, _stringify, _splitdict - -# Verify if Tk is new enough to not need the Tile package -_REQUIRE_TILE = True if tkinter.TkVersion < 8.5 else False - -def _load_tile(master): - if _REQUIRE_TILE: - import os - tilelib = os.environ.get('TILE_LIBRARY') - if tilelib: - # append custom tile path to the list of directories that - # Tcl uses when attempting to resolve packages with the package - # command - master.tk.eval( - 'global auto_path; ' - 'lappend auto_path {%s}' % tilelib) - - master.tk.eval('package require tile') # TclError may be raised here - master._tile_loaded = True - -def _format_optvalue(value, script=False): - """Internal function.""" - if script: - # if caller passes a Tcl script to tk.call, all the values need to - # be grouped into words (arguments to a command in Tcl dialect) - value = _stringify(value) - elif isinstance(value, (list, tuple)): - value = _join(value) - return value - -def _format_optdict(optdict, script=False, ignore=None): - """Formats optdict to a tuple to pass it to tk.call. - - E.g. (script=False): - {'foreground': 'blue', 'padding': [1, 2, 3, 4]} returns: - ('-foreground', 'blue', '-padding', '1 2 3 4')""" - - opts = [] - for opt, value in optdict.items(): - if not ignore or opt not in ignore: - opts.append("-%s" % opt) - if value is not None: - opts.append(_format_optvalue(value, script)) - - return _flatten(opts) - -def _mapdict_values(items): - # each value in mapdict is expected to be a sequence, where each item - # is another sequence containing a state (or several) and a value - # E.g. (script=False): - # [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])] - # returns: - # ['active selected', 'grey', 'focus', [1, 2, 3, 4]] - opt_val = [] - for *state, val in items: - # hacks for backward compatibility - state[0] # raise IndexError if empty - if len(state) == 1: - # if it is empty (something that evaluates to False), then - # format it to Tcl code to denote the "normal" state - state = state[0] or '' - else: - # group multiple states - state = ' '.join(state) # raise TypeError if not str - opt_val.append(state) - if val is not None: - opt_val.append(val) - return opt_val - -def _format_mapdict(mapdict, script=False): - """Formats mapdict to pass it to tk.call. - - E.g. (script=False): - {'expand': [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])]} - - returns: - - ('-expand', '{active selected} grey focus {1, 2, 3, 4}')""" - - opts = [] - for opt, value in mapdict.items(): - opts.extend(("-%s" % opt, - _format_optvalue(_mapdict_values(value), script))) - - return _flatten(opts) - -def _format_elemcreate(etype, script=False, *args, **kw): - """Formats args and kw according to the given element factory etype.""" - spec = None - opts = () - if etype in ("image", "vsapi"): - if etype == "image": # define an element based on an image - # first arg should be the default image name - iname = args[0] - # next args, if any, are statespec/value pairs which is almost - # a mapdict, but we just need the value - imagespec = _join(_mapdict_values(args[1:])) - spec = "%s %s" % (iname, imagespec) - - else: - # define an element whose visual appearance is drawn using the - # Microsoft Visual Styles API which is responsible for the - # themed styles on Windows XP and Vista. - # Availability: Tk 8.6, Windows XP and Vista. - class_name, part_id = args[:2] - statemap = _join(_mapdict_values(args[2:])) - spec = "%s %s %s" % (class_name, part_id, statemap) - - opts = _format_optdict(kw, script) - - elif etype == "from": # clone an element - # it expects a themename and optionally an element to clone from, - # otherwise it will clone {} (empty element) - spec = args[0] # theme name - if len(args) > 1: # elementfrom specified - opts = (_format_optvalue(args[1], script),) - - if script: - spec = '{%s}' % spec - opts = ' '.join(opts) - - return spec, opts - -def _format_layoutlist(layout, indent=0, indent_size=2): - """Formats a layout list so we can pass the result to ttk::style - layout and ttk::style settings. Note that the layout doesn't have to - be a list necessarily. - - E.g.: - [("Menubutton.background", None), - ("Menubutton.button", {"children": - [("Menubutton.focus", {"children": - [("Menubutton.padding", {"children": - [("Menubutton.label", {"side": "left", "expand": 1})] - })] - })] - }), - ("Menubutton.indicator", {"side": "right"}) - ] - - returns: - - Menubutton.background - Menubutton.button -children { - Menubutton.focus -children { - Menubutton.padding -children { - Menubutton.label -side left -expand 1 - } - } - } - Menubutton.indicator -side right""" - script = [] - - for layout_elem in layout: - elem, opts = layout_elem - opts = opts or {} - fopts = ' '.join(_format_optdict(opts, True, ("children",))) - head = "%s%s%s" % (' ' * indent, elem, (" %s" % fopts) if fopts else '') - - if "children" in opts: - script.append(head + " -children {") - indent += indent_size - newscript, indent = _format_layoutlist(opts['children'], indent, - indent_size) - script.append(newscript) - indent -= indent_size - script.append('%s}' % (' ' * indent)) - else: - script.append(head) - - return '\n'.join(script), indent - -def _script_from_settings(settings): - """Returns an appropriate script, based on settings, according to - theme_settings definition to be used by theme_settings and - theme_create.""" - script = [] - # a script will be generated according to settings passed, which - # will then be evaluated by Tcl - for name, opts in settings.items(): - # will format specific keys according to Tcl code - if opts.get('configure'): # format 'configure' - s = ' '.join(_format_optdict(opts['configure'], True)) - script.append("ttk::style configure %s %s;" % (name, s)) - - if opts.get('map'): # format 'map' - s = ' '.join(_format_mapdict(opts['map'], True)) - script.append("ttk::style map %s %s;" % (name, s)) - - if 'layout' in opts: # format 'layout' which may be empty - if not opts['layout']: - s = 'null' # could be any other word, but this one makes sense - else: - s, _ = _format_layoutlist(opts['layout']) - script.append("ttk::style layout %s {\n%s\n}" % (name, s)) - - if opts.get('element create'): # format 'element create' - eopts = opts['element create'] - etype = eopts[0] - - # find where args end, and where kwargs start - argc = 1 # etype was the first one - while argc < len(eopts) and not hasattr(eopts[argc], 'items'): - argc += 1 - - elemargs = eopts[1:argc] - elemkw = eopts[argc] if argc < len(eopts) and eopts[argc] else {} - spec, opts = _format_elemcreate(etype, True, *elemargs, **elemkw) - - script.append("ttk::style element create %s %s %s %s" % ( - name, etype, spec, opts)) - - return '\n'.join(script) - -def _list_from_statespec(stuple): - """Construct a list from the given statespec tuple according to the - accepted statespec accepted by _format_mapdict.""" - nval = [] - for val in stuple: - typename = getattr(val, 'typename', None) - if typename is None: - nval.append(val) - else: # this is a Tcl object - val = str(val) - if typename == 'StateSpec': - val = val.split() - nval.append(val) - - it = iter(nval) - return [_flatten(spec) for spec in zip(it, it)] - -def _list_from_layouttuple(tk, ltuple): - """Construct a list from the tuple returned by ttk::layout, this is - somewhat the reverse of _format_layoutlist.""" - ltuple = tk.splitlist(ltuple) - res = [] - - indx = 0 - while indx < len(ltuple): - name = ltuple[indx] - opts = {} - res.append((name, opts)) - indx += 1 - - while indx < len(ltuple): # grab name's options - opt, val = ltuple[indx:indx + 2] - if not opt.startswith('-'): # found next name - break - - opt = opt[1:] # remove the '-' from the option - indx += 2 - - if opt == 'children': - val = _list_from_layouttuple(tk, val) - - opts[opt] = val - - return res - -def _val_or_dict(tk, options, *args): - """Format options then call Tk command with args and options and return - the appropriate result. - - If no option is specified, a dict is returned. If an option is - specified with the None value, the value for that option is returned. - Otherwise, the function just sets the passed options and the caller - shouldn't be expecting a return value anyway.""" - options = _format_optdict(options) - res = tk.call(*(args + options)) - - if len(options) % 2: # option specified without a value, return its value - return res - - return _splitdict(tk, res, conv=_tclobj_to_py) - -def _convert_stringval(value): - """Converts a value to, hopefully, a more appropriate Python object.""" - value = str(value) - try: - value = int(value) - except (ValueError, TypeError): - pass - - return value - -def _to_number(x): - if isinstance(x, str): - if '.' in x: - x = float(x) - else: - x = int(x) - return x - -def _tclobj_to_py(val): - """Return value converted from Tcl object to Python object.""" - if val and hasattr(val, '__len__') and not isinstance(val, str): - if getattr(val[0], 'typename', None) == 'StateSpec': - val = _list_from_statespec(val) - else: - val = list(map(_convert_stringval, val)) - - elif hasattr(val, 'typename'): # some other (single) Tcl object - val = _convert_stringval(val) - - return val - -def tclobjs_to_py(adict): - """Returns adict with its values converted from Tcl objects to Python - objects.""" - for opt, val in adict.items(): - adict[opt] = _tclobj_to_py(val) - - return adict - -def setup_master(master=None): - """If master is not None, itself is returned. If master is None, - the default master is returned if there is one, otherwise a new - master is created and returned. - - If it is not allowed to use the default root and master is None, - RuntimeError is raised.""" - if master is None: - if tkinter._support_default_root: - master = tkinter._default_root or tkinter.Tk() - else: - raise RuntimeError( - "No master specified and tkinter is " - "configured to not support default root") - return master - - -class Style(object): - """Manipulate style database.""" - - _name = "ttk::style" - - def __init__(self, master=None): - master = setup_master(master) - - if not getattr(master, '_tile_loaded', False): - # Load tile now, if needed - _load_tile(master) - - self.master = master - self.tk = self.master.tk - - - def configure(self, style, query_opt=None, **kw): - """Query or sets the default value of the specified option(s) in - style. - - Each key in kw is an option and each value is either a string or - a sequence identifying the value for that option.""" - if query_opt is not None: - kw[query_opt] = None - result = _val_or_dict(self.tk, kw, self._name, "configure", style) - if result or query_opt: - return result - - - def map(self, style, query_opt=None, **kw): - """Query or sets dynamic values of the specified option(s) in - style. - - Each key in kw is an option and each value should be a list or a - tuple (usually) containing statespecs grouped in tuples, or list, - or something else of your preference. A statespec is compound of - one or more states and then a value.""" - if query_opt is not None: - return _list_from_statespec(self.tk.splitlist( - self.tk.call(self._name, "map", style, '-%s' % query_opt))) - - return _splitdict( - self.tk, - self.tk.call(self._name, "map", style, *_format_mapdict(kw)), - conv=_tclobj_to_py) - - - def lookup(self, style, option, state=None, default=None): - """Returns the value specified for option in style. - - If state is specified it is expected to be a sequence of one - or more states. If the default argument is set, it is used as - a fallback value in case no specification for option is found.""" - state = ' '.join(state) if state else '' - - return self.tk.call(self._name, "lookup", style, '-%s' % option, - state, default) - - - def layout(self, style, layoutspec=None): - """Define the widget layout for given style. If layoutspec is - omitted, return the layout specification for given style. - - layoutspec is expected to be a list or an object different than - None that evaluates to False if you want to "turn off" that style. - If it is a list (or tuple, or something else), each item should be - a tuple where the first item is the layout name and the second item - should have the format described below: - - LAYOUTS - - A layout can contain the value None, if takes no options, or - a dict of options specifying how to arrange the element. - The layout mechanism uses a simplified version of the pack - geometry manager: given an initial cavity, each element is - allocated a parcel. Valid options/values are: - - side: whichside - Specifies which side of the cavity to place the - element; one of top, right, bottom or left. If - omitted, the element occupies the entire cavity. - - sticky: nswe - Specifies where the element is placed inside its - allocated parcel. - - children: [sublayout... ] - Specifies a list of elements to place inside the - element. Each element is a tuple (or other sequence) - where the first item is the layout name, and the other - is a LAYOUT.""" - lspec = None - if layoutspec: - lspec = _format_layoutlist(layoutspec)[0] - elif layoutspec is not None: # will disable the layout ({}, '', etc) - lspec = "null" # could be any other word, but this may make sense - # when calling layout(style) later - - return _list_from_layouttuple(self.tk, - self.tk.call(self._name, "layout", style, lspec)) - - - def element_create(self, elementname, etype, *args, **kw): - """Create a new element in the current theme of given etype.""" - spec, opts = _format_elemcreate(etype, False, *args, **kw) - self.tk.call(self._name, "element", "create", elementname, etype, - spec, *opts) - - - def element_names(self): - """Returns the list of elements defined in the current theme.""" - return tuple(n.lstrip('-') for n in self.tk.splitlist( - self.tk.call(self._name, "element", "names"))) - - - def element_options(self, elementname): - """Return the list of elementname's options.""" - return tuple(o.lstrip('-') for o in self.tk.splitlist( - self.tk.call(self._name, "element", "options", elementname))) - - - def theme_create(self, themename, parent=None, settings=None): - """Creates a new theme. - - It is an error if themename already exists. If parent is - specified, the new theme will inherit styles, elements and - layouts from the specified parent theme. If settings are present, - they are expected to have the same syntax used for theme_settings.""" - script = _script_from_settings(settings) if settings else '' - - if parent: - self.tk.call(self._name, "theme", "create", themename, - "-parent", parent, "-settings", script) - else: - self.tk.call(self._name, "theme", "create", themename, - "-settings", script) - - - def theme_settings(self, themename, settings): - """Temporarily sets the current theme to themename, apply specified - settings and then restore the previous theme. - - Each key in settings is a style and each value may contain the - keys 'configure', 'map', 'layout' and 'element create' and they - are expected to have the same format as specified by the methods - configure, map, layout and element_create respectively.""" - script = _script_from_settings(settings) - self.tk.call(self._name, "theme", "settings", themename, script) - - - def theme_names(self): - """Returns a list of all known themes.""" - return self.tk.splitlist(self.tk.call(self._name, "theme", "names")) - - - def theme_use(self, themename=None): - """If themename is None, returns the theme in use, otherwise, set - the current theme to themename, refreshes all widgets and emits - a <<ThemeChanged>> event.""" - if themename is None: - # Starting on Tk 8.6, checking this global is no longer needed - # since it allows doing self.tk.call(self._name, "theme", "use") - return self.tk.eval("return $ttk::currentTheme") - - # using "ttk::setTheme" instead of "ttk::style theme use" causes - # the variable currentTheme to be updated, also, ttk::setTheme calls - # "ttk::style theme use" in order to change theme. - self.tk.call("ttk::setTheme", themename) - - -class Widget(tkinter.Widget): - """Base class for Tk themed widgets.""" - - def __init__(self, master, widgetname, kw=None): - """Constructs a Ttk Widget with the parent master. - - STANDARD OPTIONS - - class, cursor, takefocus, style - - SCROLLABLE WIDGET OPTIONS - - xscrollcommand, yscrollcommand - - LABEL WIDGET OPTIONS - - text, textvariable, underline, image, compound, width - - WIDGET STATES - - active, disabled, focus, pressed, selected, background, - readonly, alternate, invalid - """ - master = setup_master(master) - if not getattr(master, '_tile_loaded', False): - # Load tile now, if needed - _load_tile(master) - tkinter.Widget.__init__(self, master, widgetname, kw=kw) - - - def identify(self, x, y): - """Returns the name of the element at position x, y, or the empty - string if the point does not lie within any element. - - x and y are pixel coordinates relative to the widget.""" - return self.tk.call(self._w, "identify", x, y) - - - def instate(self, statespec, callback=None, *args, **kw): - """Test the widget's state. - - If callback is not specified, returns True if the widget state - matches statespec and False otherwise. If callback is specified, - then it will be invoked with *args, **kw if the widget state - matches statespec. statespec is expected to be a sequence.""" - ret = self.tk.getboolean( - self.tk.call(self._w, "instate", ' '.join(statespec))) - if ret and callback: - return callback(*args, **kw) - - return ret - - - def state(self, statespec=None): - """Modify or inquire widget state. - - Widget state is returned if statespec is None, otherwise it is - set according to the statespec flags and then a new state spec - is returned indicating which flags were changed. statespec is - expected to be a sequence.""" - if statespec is not None: - statespec = ' '.join(statespec) - - return self.tk.splitlist(str(self.tk.call(self._w, "state", statespec))) - - -class Button(Widget): - """Ttk Button widget, displays a textual label and/or image, and - evaluates a command when pressed.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Button widget with the parent master. - - STANDARD OPTIONS - - class, compound, cursor, image, state, style, takefocus, - text, textvariable, underline, width - - WIDGET-SPECIFIC OPTIONS - - command, default, width - """ - Widget.__init__(self, master, "ttk::button", kw) - - - def invoke(self): - """Invokes the command associated with the button.""" - return self.tk.call(self._w, "invoke") - - -class Checkbutton(Widget): - """Ttk Checkbutton widget which is either in on- or off-state.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Checkbutton widget with the parent master. - - STANDARD OPTIONS - - class, compound, cursor, image, state, style, takefocus, - text, textvariable, underline, width - - WIDGET-SPECIFIC OPTIONS - - command, offvalue, onvalue, variable - """ - Widget.__init__(self, master, "ttk::checkbutton", kw) - - - def invoke(self): - """Toggles between the selected and deselected states and - invokes the associated command. If the widget is currently - selected, sets the option variable to the offvalue option - and deselects the widget; otherwise, sets the option variable - to the option onvalue. - - Returns the result of the associated command.""" - return self.tk.call(self._w, "invoke") - - -class Entry(Widget, tkinter.Entry): - """Ttk Entry widget displays a one-line text string and allows that - string to be edited by the user.""" - - def __init__(self, master=None, widget=None, **kw): - """Constructs a Ttk Entry widget with the parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus, xscrollcommand - - WIDGET-SPECIFIC OPTIONS - - exportselection, invalidcommand, justify, show, state, - textvariable, validate, validatecommand, width - - VALIDATION MODES - - none, key, focus, focusin, focusout, all - """ - Widget.__init__(self, master, widget or "ttk::entry", kw) - - - def bbox(self, index): - """Return a tuple of (x, y, width, height) which describes the - bounding box of the character given by index.""" - return self._getints(self.tk.call(self._w, "bbox", index)) - - - def identify(self, x, y): - """Returns the name of the element at position x, y, or the - empty string if the coordinates are outside the window.""" - return self.tk.call(self._w, "identify", x, y) - - - def validate(self): - """Force revalidation, independent of the conditions specified - by the validate option. Returns False if validation fails, True - if it succeeds. Sets or clears the invalid state accordingly.""" - return self.tk.getboolean(self.tk.call(self._w, "validate")) - - -class Combobox(Entry): - """Ttk Combobox widget combines a text field with a pop-down list of - values.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Combobox widget with the parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - exportselection, justify, height, postcommand, state, - textvariable, values, width - """ - Entry.__init__(self, master, "ttk::combobox", **kw) - - - def current(self, newindex=None): - """If newindex is supplied, sets the combobox value to the - element at position newindex in the list of values. Otherwise, - returns the index of the current value in the list of values - or -1 if the current value does not appear in the list.""" - if newindex is None: - return self.tk.getint(self.tk.call(self._w, "current")) - return self.tk.call(self._w, "current", newindex) - - - def set(self, value): - """Sets the value of the combobox to value.""" - self.tk.call(self._w, "set", value) - - -class Frame(Widget): - """Ttk Frame widget is a container, used to group other widgets - together.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Frame with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - borderwidth, relief, padding, width, height - """ - Widget.__init__(self, master, "ttk::frame", kw) - - -class Label(Widget): - """Ttk Label widget displays a textual label and/or image.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Label with parent master. - - STANDARD OPTIONS - - class, compound, cursor, image, style, takefocus, text, - textvariable, underline, width - - WIDGET-SPECIFIC OPTIONS - - anchor, background, font, foreground, justify, padding, - relief, text, wraplength - """ - Widget.__init__(self, master, "ttk::label", kw) - - -class Labelframe(Widget): - """Ttk Labelframe widget is a container used to group other widgets - together. It has an optional label, which may be a plain text string - or another widget.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Labelframe with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - labelanchor, text, underline, padding, labelwidget, width, - height - """ - Widget.__init__(self, master, "ttk::labelframe", kw) - -LabelFrame = Labelframe # tkinter name compatibility - - -class Menubutton(Widget): - """Ttk Menubutton widget displays a textual label and/or image, and - displays a menu when pressed.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Menubutton with parent master. - - STANDARD OPTIONS - - class, compound, cursor, image, state, style, takefocus, - text, textvariable, underline, width - - WIDGET-SPECIFIC OPTIONS - - direction, menu - """ - Widget.__init__(self, master, "ttk::menubutton", kw) - - -class Notebook(Widget): - """Ttk Notebook widget manages a collection of windows and displays - a single one at a time. Each child window is associated with a tab, - which the user may select to change the currently-displayed window.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Notebook with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - height, padding, width - - TAB OPTIONS - - state, sticky, padding, text, image, compound, underline - - TAB IDENTIFIERS (tab_id) - - The tab_id argument found in several methods may take any of - the following forms: - - * An integer between zero and the number of tabs - * The name of a child window - * A positional specification of the form "@x,y", which - defines the tab - * The string "current", which identifies the - currently-selected tab - * The string "end", which returns the number of tabs (only - valid for method index) - """ - Widget.__init__(self, master, "ttk::notebook", kw) - - - def add(self, child, **kw): - """Adds a new tab to the notebook. - - If window is currently managed by the notebook but hidden, it is - restored to its previous position.""" - self.tk.call(self._w, "add", child, *(_format_optdict(kw))) - - - def forget(self, tab_id): - """Removes the tab specified by tab_id, unmaps and unmanages the - associated window.""" - self.tk.call(self._w, "forget", tab_id) - - - def hide(self, tab_id): - """Hides the tab specified by tab_id. - - The tab will not be displayed, but the associated window remains - managed by the notebook and its configuration remembered. Hidden - tabs may be restored with the add command.""" - self.tk.call(self._w, "hide", tab_id) - - - def identify(self, x, y): - """Returns the name of the tab element at position x, y, or the - empty string if none.""" - return self.tk.call(self._w, "identify", x, y) - - - def index(self, tab_id): - """Returns the numeric index of the tab specified by tab_id, or - the total number of tabs if tab_id is the string "end".""" - return self.tk.getint(self.tk.call(self._w, "index", tab_id)) - - - def insert(self, pos, child, **kw): - """Inserts a pane at the specified position. - - pos is either the string end, an integer index, or the name of - a managed child. If child is already managed by the notebook, - moves it to the specified position.""" - self.tk.call(self._w, "insert", pos, child, *(_format_optdict(kw))) - - - def select(self, tab_id=None): - """Selects the specified tab. - - The associated child window will be displayed, and the - previously-selected window (if different) is unmapped. If tab_id - is omitted, returns the widget name of the currently selected - pane.""" - return self.tk.call(self._w, "select", tab_id) - - - def tab(self, tab_id, option=None, **kw): - """Query or modify the options of the specific tab_id. - - If kw is not given, returns a dict of the tab option values. If option - is specified, returns the value of that option. Otherwise, sets the - options to the corresponding values.""" - if option is not None: - kw[option] = None - return _val_or_dict(self.tk, kw, self._w, "tab", tab_id) - - - def tabs(self): - """Returns a list of windows managed by the notebook.""" - return self.tk.splitlist(self.tk.call(self._w, "tabs") or ()) - - - def enable_traversal(self): - """Enable keyboard traversal for a toplevel window containing - this notebook. - - This will extend the bindings for the toplevel window containing - this notebook as follows: - - Control-Tab: selects the tab following the currently selected - one - - Shift-Control-Tab: selects the tab preceding the currently - selected one - - Alt-K: where K is the mnemonic (underlined) character of any - tab, will select that tab. - - Multiple notebooks in a single toplevel may be enabled for - traversal, including nested notebooks. However, notebook traversal - only works properly if all panes are direct children of the - notebook.""" - # The only, and good, difference I see is about mnemonics, which works - # after calling this method. Control-Tab and Shift-Control-Tab always - # works (here at least). - self.tk.call("ttk::notebook::enableTraversal", self._w) - - -class Panedwindow(Widget, tkinter.PanedWindow): - """Ttk Panedwindow widget displays a number of subwindows, stacked - either vertically or horizontally.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Panedwindow with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - orient, width, height - - PANE OPTIONS - - weight - """ - Widget.__init__(self, master, "ttk::panedwindow", kw) - - - forget = tkinter.PanedWindow.forget # overrides Pack.forget - - - def insert(self, pos, child, **kw): - """Inserts a pane at the specified positions. - - pos is either the string end, and integer index, or the name - of a child. If child is already managed by the paned window, - moves it to the specified position.""" - self.tk.call(self._w, "insert", pos, child, *(_format_optdict(kw))) - - - def pane(self, pane, option=None, **kw): - """Query or modify the options of the specified pane. - - pane is either an integer index or the name of a managed subwindow. - If kw is not given, returns a dict of the pane option values. If - option is specified then the value for that option is returned. - Otherwise, sets the options to the corresponding values.""" - if option is not None: - kw[option] = None - return _val_or_dict(self.tk, kw, self._w, "pane", pane) - - - def sashpos(self, index, newpos=None): - """If newpos is specified, sets the position of sash number index. - - May adjust the positions of adjacent sashes to ensure that - positions are monotonically increasing. Sash positions are further - constrained to be between 0 and the total size of the widget. - - Returns the new position of sash number index.""" - return self.tk.getint(self.tk.call(self._w, "sashpos", index, newpos)) - -PanedWindow = Panedwindow # tkinter name compatibility - - -class Progressbar(Widget): - """Ttk Progressbar widget shows the status of a long-running - operation. They can operate in two modes: determinate mode shows the - amount completed relative to the total amount of work to be done, and - indeterminate mode provides an animated display to let the user know - that something is happening.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Progressbar with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - orient, length, mode, maximum, value, variable, phase - """ - Widget.__init__(self, master, "ttk::progressbar", kw) - - - def start(self, interval=None): - """Begin autoincrement mode: schedules a recurring timer event - that calls method step every interval milliseconds. - - interval defaults to 50 milliseconds (20 steps/second) if omitted.""" - self.tk.call(self._w, "start", interval) - - - def step(self, amount=None): - """Increments the value option by amount. - - amount defaults to 1.0 if omitted.""" - self.tk.call(self._w, "step", amount) - - - def stop(self): - """Stop autoincrement mode: cancels any recurring timer event - initiated by start.""" - self.tk.call(self._w, "stop") - - -class Radiobutton(Widget): - """Ttk Radiobutton widgets are used in groups to show or change a - set of mutually-exclusive options.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Radiobutton with parent master. - - STANDARD OPTIONS - - class, compound, cursor, image, state, style, takefocus, - text, textvariable, underline, width - - WIDGET-SPECIFIC OPTIONS - - command, value, variable - """ - Widget.__init__(self, master, "ttk::radiobutton", kw) - - - def invoke(self): - """Sets the option variable to the option value, selects the - widget, and invokes the associated command. - - Returns the result of the command, or an empty string if - no command is specified.""" - return self.tk.call(self._w, "invoke") - - -class Scale(Widget, tkinter.Scale): - """Ttk Scale widget is typically used to control the numeric value of - a linked variable that varies uniformly over some range.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Scale with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - command, from, length, orient, to, value, variable - """ - Widget.__init__(self, master, "ttk::scale", kw) - - - def configure(self, cnf=None, **kw): - """Modify or query scale options. - - Setting a value for any of the "from", "from_" or "to" options - generates a <<RangeChanged>> event.""" - if cnf: - kw.update(cnf) - Widget.configure(self, **kw) - if any(['from' in kw, 'from_' in kw, 'to' in kw]): - self.event_generate('<<RangeChanged>>') - - - def get(self, x=None, y=None): - """Get the current value of the value option, or the value - corresponding to the coordinates x, y if they are specified. - - x and y are pixel coordinates relative to the scale widget - origin.""" - return self.tk.call(self._w, 'get', x, y) - - -class Scrollbar(Widget, tkinter.Scrollbar): - """Ttk Scrollbar controls the viewport of a scrollable widget.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Scrollbar with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - command, orient - """ - Widget.__init__(self, master, "ttk::scrollbar", kw) - - -class Separator(Widget): - """Ttk Separator widget displays a horizontal or vertical separator - bar.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Separator with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus - - WIDGET-SPECIFIC OPTIONS - - orient - """ - Widget.__init__(self, master, "ttk::separator", kw) - - -class Sizegrip(Widget): - """Ttk Sizegrip allows the user to resize the containing toplevel - window by pressing and dragging the grip.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Sizegrip with parent master. - - STANDARD OPTIONS - - class, cursor, state, style, takefocus - """ - Widget.__init__(self, master, "ttk::sizegrip", kw) - - -class Spinbox(Entry): - """Ttk Spinbox is an Entry with increment and decrement arrows - - It is commonly used for number entry or to select from a list of - string values. - """ - - def __init__(self, master=None, **kw): - """Construct a Ttk Spinbox widget with the parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus, validate, - validatecommand, xscrollcommand, invalidcommand - - WIDGET-SPECIFIC OPTIONS - - to, from_, increment, values, wrap, format, command - """ - Entry.__init__(self, master, "ttk::spinbox", **kw) - - - def set(self, value): - """Sets the value of the Spinbox to value.""" - self.tk.call(self._w, "set", value) - - -class Treeview(Widget, tkinter.XView, tkinter.YView): - """Ttk Treeview widget displays a hierarchical collection of items. - - Each item has a textual label, an optional image, and an optional list - of data values. The data values are displayed in successive columns - after the tree label.""" - - def __init__(self, master=None, **kw): - """Construct a Ttk Treeview with parent master. - - STANDARD OPTIONS - - class, cursor, style, takefocus, xscrollcommand, - yscrollcommand - - WIDGET-SPECIFIC OPTIONS - - columns, displaycolumns, height, padding, selectmode, show - - ITEM OPTIONS - - text, image, values, open, tags - - TAG OPTIONS - - foreground, background, font, image - """ - Widget.__init__(self, master, "ttk::treeview", kw) - - - def bbox(self, item, column=None): - """Returns the bounding box (relative to the treeview widget's - window) of the specified item in the form x y width height. - - If column is specified, returns the bounding box of that cell. - If the item is not visible (i.e., if it is a descendant of a - closed item or is scrolled offscreen), returns an empty string.""" - return self._getints(self.tk.call(self._w, "bbox", item, column)) or '' - - - def get_children(self, item=None): - """Returns a tuple of children belonging to item. - - If item is not specified, returns root children.""" - return self.tk.splitlist( - self.tk.call(self._w, "children", item or '') or ()) - - - def set_children(self, item, *newchildren): - """Replaces item's child with newchildren. - - Children present in item that are not present in newchildren - are detached from tree. No items in newchildren may be an - ancestor of item.""" - self.tk.call(self._w, "children", item, newchildren) - - - def column(self, column, option=None, **kw): - """Query or modify the options for the specified column. - - If kw is not given, returns a dict of the column option values. If - option is specified then the value for that option is returned. - Otherwise, sets the options to the corresponding values.""" - if option is not None: - kw[option] = None - return _val_or_dict(self.tk, kw, self._w, "column", column) - - - def delete(self, *items): - """Delete all specified items and all their descendants. The root - item may not be deleted.""" - self.tk.call(self._w, "delete", items) - - - def detach(self, *items): - """Unlinks all of the specified items from the tree. - - The items and all of their descendants are still present, and may - be reinserted at another point in the tree, but will not be - displayed. The root item may not be detached.""" - self.tk.call(self._w, "detach", items) - - - def exists(self, item): - """Returns True if the specified item is present in the tree, - False otherwise.""" - return self.tk.getboolean(self.tk.call(self._w, "exists", item)) - - - def focus(self, item=None): - """If item is specified, sets the focus item to item. Otherwise, - returns the current focus item, or '' if there is none.""" - return self.tk.call(self._w, "focus", item) - - - def heading(self, column, option=None, **kw): - """Query or modify the heading options for the specified column. - - If kw is not given, returns a dict of the heading option values. If - option is specified then the value for that option is returned. - Otherwise, sets the options to the corresponding values. - - Valid options/values are: - text: text - The text to display in the column heading - image: image_name - Specifies an image to display to the right of the column - heading - anchor: anchor - Specifies how the heading text should be aligned. One of - the standard Tk anchor values - command: callback - A callback to be invoked when the heading label is - pressed. - - To configure the tree column heading, call this with column = "#0" """ - cmd = kw.get('command') - if cmd and not isinstance(cmd, str): - # callback not registered yet, do it now - kw['command'] = self.master.register(cmd, self._substitute) - - if option is not None: - kw[option] = None - - return _val_or_dict(self.tk, kw, self._w, 'heading', column) - - - def identify(self, component, x, y): - """Returns a description of the specified component under the - point given by x and y, or the empty string if no such component - is present at that position.""" - return self.tk.call(self._w, "identify", component, x, y) - - - def identify_row(self, y): - """Returns the item ID of the item at position y.""" - return self.identify("row", 0, y) - - - def identify_column(self, x): - """Returns the data column identifier of the cell at position x. - - The tree column has ID #0.""" - return self.identify("column", x, 0) - - - def identify_region(self, x, y): - """Returns one of: - - heading: Tree heading area. - separator: Space between two columns headings; - tree: The tree area. - cell: A data cell. - - * Availability: Tk 8.6""" - return self.identify("region", x, y) - - - def identify_element(self, x, y): - """Returns the element at position x, y. - - * Availability: Tk 8.6""" - return self.identify("element", x, y) - - - def index(self, item): - """Returns the integer index of item within its parent's list - of children.""" - return self.tk.getint(self.tk.call(self._w, "index", item)) - - - def insert(self, parent, index, iid=None, **kw): - """Creates a new item and return the item identifier of the newly - created item. - - parent is the item ID of the parent item, or the empty string - to create a new top-level item. index is an integer, or the value - end, specifying where in the list of parent's children to insert - the new item. If index is less than or equal to zero, the new node - is inserted at the beginning, if index is greater than or equal to - the current number of children, it is inserted at the end. If iid - is specified, it is used as the item identifier, iid must not - already exist in the tree. Otherwise, a new unique identifier - is generated.""" - opts = _format_optdict(kw) - if iid is not None: - res = self.tk.call(self._w, "insert", parent, index, - "-id", iid, *opts) - else: - res = self.tk.call(self._w, "insert", parent, index, *opts) - - return res - - - def item(self, item, option=None, **kw): - """Query or modify the options for the specified item. - - If no options are given, a dict with options/values for the item - is returned. If option is specified then the value for that option - is returned. Otherwise, sets the options to the corresponding - values as given by kw.""" - if option is not None: - kw[option] = None - return _val_or_dict(self.tk, kw, self._w, "item", item) - - - def move(self, item, parent, index): - """Moves item to position index in parent's list of children. - - It is illegal to move an item under one of its descendants. If - index is less than or equal to zero, item is moved to the - beginning, if greater than or equal to the number of children, - it is moved to the end. If item was detached it is reattached.""" - self.tk.call(self._w, "move", item, parent, index) - - reattach = move # A sensible method name for reattaching detached items - - - def next(self, item): - """Returns the identifier of item's next sibling, or '' if item - is the last child of its parent.""" - return self.tk.call(self._w, "next", item) - - - def parent(self, item): - """Returns the ID of the parent of item, or '' if item is at the - top level of the hierarchy.""" - return self.tk.call(self._w, "parent", item) - - - def prev(self, item): - """Returns the identifier of item's previous sibling, or '' if - item is the first child of its parent.""" - return self.tk.call(self._w, "prev", item) - - - def see(self, item): - """Ensure that item is visible. - - Sets all of item's ancestors open option to True, and scrolls - the widget if necessary so that item is within the visible - portion of the tree.""" - self.tk.call(self._w, "see", item) - - - def selection(self): - """Returns the tuple of selected items.""" - return self.tk.splitlist(self.tk.call(self._w, "selection")) - - - def _selection(self, selop, items): - if len(items) == 1 and isinstance(items[0], (tuple, list)): - items = items[0] - - self.tk.call(self._w, "selection", selop, items) - - - def selection_set(self, *items): - """The specified items becomes the new selection.""" - self._selection("set", items) - - - def selection_add(self, *items): - """Add all of the specified items to the selection.""" - self._selection("add", items) - - - def selection_remove(self, *items): - """Remove all of the specified items from the selection.""" - self._selection("remove", items) - - - def selection_toggle(self, *items): - """Toggle the selection state of each specified item.""" - self._selection("toggle", items) - - - def set(self, item, column=None, value=None): - """Query or set the value of given item. - - With one argument, return a dictionary of column/value pairs - for the specified item. With two arguments, return the current - value of the specified column. With three arguments, set the - value of given column in given item to the specified value.""" - res = self.tk.call(self._w, "set", item, column, value) - if column is None and value is None: - return _splitdict(self.tk, res, - cut_minus=False, conv=_tclobj_to_py) - else: - return res - - - def tag_bind(self, tagname, sequence=None, callback=None): - """Bind a callback for the given event sequence to the tag tagname. - When an event is delivered to an item, the callbacks for each - of the item's tags option are called.""" - self._bind((self._w, "tag", "bind", tagname), sequence, callback, add=0) - - - def tag_configure(self, tagname, option=None, **kw): - """Query or modify the options for the specified tagname. - - If kw is not given, returns a dict of the option settings for tagname. - If option is specified, returns the value for that option for the - specified tagname. Otherwise, sets the options to the corresponding - values for the given tagname.""" - if option is not None: - kw[option] = None - return _val_or_dict(self.tk, kw, self._w, "tag", "configure", - tagname) - - - def tag_has(self, tagname, item=None): - """If item is specified, returns 1 or 0 depending on whether the - specified item has the given tagname. Otherwise, returns a list of - all items which have the specified tag. - - * Availability: Tk 8.6""" - if item is None: - return self.tk.splitlist( - self.tk.call(self._w, "tag", "has", tagname)) - else: - return self.tk.getboolean( - self.tk.call(self._w, "tag", "has", tagname, item)) - - -# Extensions - -class LabeledScale(Frame): - """A Ttk Scale widget with a Ttk Label widget indicating its - current value. - - The Ttk Scale can be accessed through instance.scale, and Ttk Label - can be accessed through instance.label""" - - def __init__(self, master=None, variable=None, from_=0, to=10, **kw): - """Construct a horizontal LabeledScale with parent master, a - variable to be associated with the Ttk Scale widget and its range. - If variable is not specified, a tkinter.IntVar is created. - - WIDGET-SPECIFIC OPTIONS - - compound: 'top' or 'bottom' - Specifies how to display the label relative to the scale. - Defaults to 'top'. - """ - self._label_top = kw.pop('compound', 'top') == 'top' - - Frame.__init__(self, master, **kw) - self._variable = variable or tkinter.IntVar(master) - self._variable.set(from_) - self._last_valid = from_ - - self.label = Label(self) - self.scale = Scale(self, variable=self._variable, from_=from_, to=to) - self.scale.bind('<<RangeChanged>>', self._adjust) - - # position scale and label according to the compound option - scale_side = 'bottom' if self._label_top else 'top' - label_side = 'top' if scale_side == 'bottom' else 'bottom' - self.scale.pack(side=scale_side, fill='x') - tmp = Label(self).pack(side=label_side) # place holder - self.label.place(anchor='n' if label_side == 'top' else 's') - - # update the label as scale or variable changes - self.__tracecb = self._variable.trace_variable('w', self._adjust) - self.bind('<Configure>', self._adjust) - self.bind('<Map>', self._adjust) - - - def destroy(self): - """Destroy this widget and possibly its associated variable.""" - try: - self._variable.trace_vdelete('w', self.__tracecb) - except AttributeError: - pass - else: - del self._variable - super().destroy() - self.label = None - self.scale = None - - - def _adjust(self, *args): - """Adjust the label position according to the scale.""" - def adjust_label(): - self.update_idletasks() # "force" scale redraw - - x, y = self.scale.coords() - if self._label_top: - y = self.scale.winfo_y() - self.label.winfo_reqheight() - else: - y = self.scale.winfo_reqheight() + self.label.winfo_reqheight() - - self.label.place_configure(x=x, y=y) - - from_ = _to_number(self.scale['from']) - to = _to_number(self.scale['to']) - if to < from_: - from_, to = to, from_ - newval = self._variable.get() - if not from_ <= newval <= to: - # value outside range, set value back to the last valid one - self.value = self._last_valid - return - - self._last_valid = newval - self.label['text'] = newval - self.after_idle(adjust_label) - - @property - def value(self): - """Return current scale value.""" - return self._variable.get() - - @value.setter - def value(self, val): - """Set new scale value.""" - self._variable.set(val) - - -class OptionMenu(Menubutton): - """Themed OptionMenu, based after tkinter's OptionMenu, which allows - the user to select a value from a menu.""" - - def __init__(self, master, variable, default=None, *values, **kwargs): - """Construct a themed OptionMenu widget with master as the parent, - the resource textvariable set to variable, the initially selected - value specified by the default parameter, the menu values given by - *values and additional keywords. - - WIDGET-SPECIFIC OPTIONS - - style: stylename - Menubutton style. - direction: 'above', 'below', 'left', 'right', or 'flush' - Menubutton direction. - command: callback - A callback that will be invoked after selecting an item. - """ - kw = {'textvariable': variable, 'style': kwargs.pop('style', None), - 'direction': kwargs.pop('direction', None)} - Menubutton.__init__(self, master, **kw) - self['menu'] = tkinter.Menu(self, tearoff=False) - - self._variable = variable - self._callback = kwargs.pop('command', None) - if kwargs: - raise tkinter.TclError('unknown option -%s' % ( - next(iter(kwargs.keys())))) - - self.set_menu(default, *values) - - - def __getitem__(self, item): - if item == 'menu': - return self.nametowidget(Menubutton.__getitem__(self, item)) - - return Menubutton.__getitem__(self, item) - - - def set_menu(self, default=None, *values): - """Build a new menu of radiobuttons with *values and optionally - a default value.""" - menu = self['menu'] - menu.delete(0, 'end') - for val in values: - menu.add_radiobutton(label=val, - command=tkinter._setit(self._variable, val, self._callback), - variable=self._variable) - - if default: - self._variable.set(default) - - - def destroy(self): - """Destroy this widget and its associated variable.""" - try: - del self._variable - except AttributeError: - pass - super().destroy() diff --git a/Lib/trace.py b/Lib/trace.py index 38a13e2..96af260 100755 --- a/Lib/trace.py +++ b/Lib/trace.py @@ -360,7 +360,7 @@ class CoverageResults: try: outfile = open(path, "w") except IOError, err: - print >> sys.stderr, ("trace: Could not open %r for writing: %s" + print >> sys.stderr, ("trace: Could not open %r for writing: %s " "- skipping" % (path, err)) return 0, 0 diff --git a/Lib/urlparse.py b/Lib/urlparse.py index 4cd3d67..f7c2b03 100644 --- a/Lib/urlparse.py +++ b/Lib/urlparse.py @@ -361,7 +361,7 @@ def unquote(s): append(item) return ''.join(res) -def parse_qs(qs, keep_blank_values=0, strict_parsing=0): +def parse_qs(qs, keep_blank_values=0, strict_parsing=0, max_num_fields=None): """Parse a query given as a string argument. Arguments: @@ -378,16 +378,20 @@ def parse_qs(qs, keep_blank_values=0, strict_parsing=0): strict_parsing: flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a ValueError exception. + + max_num_fields: int. If set, then throws a ValueError if there + are more than n fields read by parse_qsl(). """ dict = {} - for name, value in parse_qsl(qs, keep_blank_values, strict_parsing): + for name, value in parse_qsl(qs, keep_blank_values, strict_parsing, + max_num_fields): if name in dict: dict[name].append(value) else: dict[name] = [value] return dict -def parse_qsl(qs, keep_blank_values=0, strict_parsing=0): +def parse_qsl(qs, keep_blank_values=0, strict_parsing=0, max_num_fields=None): """Parse a query given as a string argument. Arguments: @@ -404,8 +408,19 @@ def parse_qsl(qs, keep_blank_values=0, strict_parsing=0): false (the default), errors are silently ignored. If true, errors raise a ValueError exception. + max_num_fields: int. If set, then throws a ValueError if there + are more than n fields read by parse_qsl(). + Returns a list, as G-d intended. """ + # If max_num_fields is defined then check that the number of fields + # is less than max_num_fields. This prevents a memory exhaustion DOS + # attack via post bodies with many fields. + if max_num_fields is not None: + num_fields = 1 + qs.count('&') + qs.count(';') + if max_num_fields < num_fields: + raise ValueError('Max number of fields exceeded') + pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')] r = [] for name_value in pairs: diff --git a/Lib/uuid.py b/Lib/uuid.py index 618bc63..80d33c0 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -419,7 +419,7 @@ def _ipconfig_getnode(): with pipe: for line in pipe: value = line.split(':')[-1].strip().lower() - if re.match('([0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]', value): + if re.match('(?:[0-9a-f][0-9a-f]-){5}[0-9a-f][0-9a-f]$', value): return int(value.replace('-', ''), 16) def _netbios_getnode(): @@ -522,6 +522,11 @@ def _random_getnode(): _node = None +_NODE_GETTERS_WIN32 = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] + +_NODE_GETTERS_UNIX = [_unixdll_getnode, _ifconfig_getnode, _arp_getnode, + _lanscan_getnode, _netstat_getnode] + def getnode(): """Get the hardware address as a 48-bit positive integer. @@ -537,18 +542,19 @@ def getnode(): import sys if sys.platform == 'win32': - getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] + getters = _NODE_GETTERS_WIN32 else: - getters = [_unixdll_getnode, _ifconfig_getnode, _arp_getnode, - _lanscan_getnode, _netstat_getnode] + getters = _NODE_GETTERS_UNIX for getter in getters + [_random_getnode]: try: _node = getter() except: continue - if _node is not None: + if (_node is not None) and (0 <= _node < (1 << 48)): return _node + assert False, '_random_getnode() returned invalid value: {}'.format(_node) + _last_timestamp = None diff --git a/Lib/webbrowser.py b/Lib/webbrowser.py index 2389179..15eeb66 100755 --- a/Lib/webbrowser.py +++ b/Lib/webbrowser.py @@ -319,11 +319,10 @@ Chromium = Chrome class Opera(UnixBrowser): "Launcher class for Opera browser." - raise_opts = ["-noraise", ""] - remote_args = ['-remote', 'openURL(%s%action)'] + remote_args = ['%action', '%s'] remote_action = "" - remote_action_newwin = ",new-window" - remote_action_newtab = ",new-page" + remote_action_newwin = "--new-window" + remote_action_newtab = "" background = True diff --git a/Lib/xml/dom/domreg.py b/Lib/xml/dom/domreg.py index ec3acdf..0835280 100644 --- a/Lib/xml/dom/domreg.py +++ b/Lib/xml/dom/domreg.py @@ -8,6 +8,8 @@ from xml.dom.minicompat import * # isinstance, StringTypes # should be published by posting to xml-sig@python.org, and are # subsequently recorded in this file. +import sys + well_known_implementations = { 'minidom':'xml.dom.minidom', '4DOM': 'xml.dom.DOMImplementation', @@ -57,7 +59,7 @@ def getDOMImplementation(name = None, features = ()): return mod.getDOMImplementation() elif name: return registered[name]() - elif "PYTHON_DOM" in os.environ: + elif not sys.flags.ignore_environment and "PYTHON_DOM" in os.environ: return getDOMImplementation(name = os.environ["PYTHON_DOM"]) # User did not specify a name, try implementations in arbitrary diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py index c30e246..05649d6 100644 --- a/Lib/xml/dom/minidom.py +++ b/Lib/xml/dom/minidom.py @@ -1273,7 +1273,7 @@ class DocumentType(Identified, Childless, Node): entity.encoding = e.encoding entity.version = e.version clone.entities._seq.append(entity) - e._call_user_data_handler(operation, n, entity) + e._call_user_data_handler(operation, e, entity) self._call_user_data_handler(operation, self, clone) return clone else: @@ -1876,7 +1876,7 @@ def _clone_node(node, deep, newOwnerDocument): entity.ownerDocument = newOwnerDocument clone.entities._seq.append(entity) if hasattr(e, '_call_user_data_handler'): - e._call_user_data_handler(operation, n, entity) + e._call_user_data_handler(operation, e, entity) else: # Note the cloning of Document and DocumentType nodes is # implementation specific. minidom handles those cases diff --git a/Lib/xml/sax/__init__.py b/Lib/xml/sax/__init__.py index 005b66e..d2e3391 100644 --- a/Lib/xml/sax/__init__.py +++ b/Lib/xml/sax/__init__.py @@ -59,7 +59,7 @@ if _false: import xml.sax.expatreader import os, sys -if "PY_SAX_PARSER" in os.environ: +if not sys.flags.ignore_environment and "PY_SAX_PARSER" in os.environ: default_parser_list = os.environ["PY_SAX_PARSER"].split(",") del os diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 3ab66ce..991a0ad 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -131,6 +131,29 @@ _CD64_NUMBER_ENTRIES_TOTAL = 7 _CD64_DIRECTORY_SIZE = 8 _CD64_OFFSET_START_CENTDIR = 9 +_DD_SIGNATURE = 0x08074b50 + +_EXTRA_FIELD_STRUCT = struct.Struct('<HH') + +def _strip_extra(extra, xids): + # Remove Extra Fields with specified IDs. + unpack = _EXTRA_FIELD_STRUCT.unpack + modified = False + buffer = [] + start = i = 0 + while i + 4 <= len(extra): + xid, xlen = unpack(extra[i : i + 4]) + j = i + 4 + xlen + if xid in xids: + if i != start: + buffer.append(extra[start : i]) + start = j + modified = True + i = j + if not modified: + return extra + return b''.join(buffer) + def _check_zipfile(fp): try: if _EndRecData(fp): @@ -1249,9 +1272,9 @@ class ZipFile(object): self.fp.write(bytes) if zinfo.flag_bits & 0x08: # Write CRC and file sizes after the file data - fmt = '<LQQ' if zip64 else '<LLL' - self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size, - zinfo.file_size)) + fmt = '<LLQQ' if zip64 else '<LLLL' + self.fp.write(struct.pack(fmt, _DD_SIGNATURE, zinfo.CRC, + zinfo.compress_size, zinfo.file_size)) self.fp.flush() self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo @@ -1293,6 +1316,7 @@ class ZipFile(object): extra_data = zinfo.extra if extra: # Append a ZIP64 field to the extra's + extra_data = _strip_extra(extra_data, (1,)) extra_data = struct.pack( '<HH' + 'Q'*len(extra), 1, 8*len(extra), *extra) + extra_data diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 7875bc8..74d1e84 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -2,6 +2,8 @@ """ This script is used to build "official" universal installers on macOS. +NEW for 3.6.8 / 2.7.16: +- also build and use Tk 8.6 for 10.6+ installers NEW for 3.6.5: - support Intel 64-bit-only () and 32-bit-only installer builds - build and link with private Tcl/Tk 8.6 for 10.9+ builds @@ -20,8 +22,8 @@ Sphinx and dependencies are installed into a venv using the python3's pip so will fetch them from PyPI if necessary. Since python3 is now used for Sphinx, build-installer.py should also be converted to use python3! -For 10.9 or greater deployment targets, build-installer builds and links -with its own copy of Tcl/Tk 8.5 and the rest of this paragraph does not +For 10.6 or greater deployment targets, build-installer builds and links +with its own copy of Tcl/Tk 8.6 and the rest of this paragraph does not apply. Otherwise, build-installer requires an installed third-party version of Tcl/Tk 8.4 (for OS X 10.4 and 10.5 deployment targets) or Tcl/TK 8.5 (for 10.6 or later) installed in /Library/Frameworks. When installed, @@ -188,9 +190,9 @@ USAGE = textwrap.dedent("""\ EXPECTED_SHARED_LIBS = {} # Are we building and linking with our own copy of Tcl/TK? -# For now, do so if deployment target is 10.9+. +# For now, do so if deployment target is 10.6+. def internalTk(): - return getDeptargetTuple() >= (10, 9) + return getDeptargetTuple() >= (10, 6) # List of names of third party software built with this installer. # The names will be inserted into the rtf version of the License. @@ -211,9 +213,9 @@ def library_recipes(): result.extend([ dict( - name="OpenSSL 1.0.2o", - url="https://www.openssl.org/source/openssl-1.0.2o.tar.gz", - checksum='44279b8557c3247cbe324e2322ecd114', + name="OpenSSL 1.0.2q", + url="https://www.openssl.org/source/openssl-1.0.2q.tar.gz", + checksum='7563e1ce046cb21948eeb6ba1a0eb71c', buildrecipe=build_universal_openssl, configure=None, install=None, @@ -824,6 +826,13 @@ def build_universal_openssl(basedir, archList): ] if no_asm: configure_opts.append("no-asm") + # OpenSSL 1.0.2o broke the Configure test for whether the compiler + # in use supports dependency rule generation (cc -M) with gcc-4.2 + # used for the 10.6+ installer builds. Patch Configure here to + # force use of "cc -M" rather than "makedepend". + runCommand( + """sed -i "" 's|my $cc_as_makedepend = 0|my $cc_as_makedepend = 1|g' Configure""") + runCommand(" ".join(["perl", "Configure"] + arch_opts[arch] + configure_opts)) runCommand("make depend") @@ -1524,12 +1533,27 @@ def buildDMG(): imagepath = imagepath + '.dmg' os.mkdir(outdir) + + # Try to mitigate race condition in certain versions of macOS, e.g. 10.9, + # when hdiutil create fails with "Resource busy". For now, just retry + # the create a few times and hope that it eventually works. + volname='Python %s'%(getFullVersion()) - runCommand("hdiutil create -format UDRW -volname %s -srcfolder %s %s"%( + cmd = ("hdiutil create -format UDRW -volname %s -srcfolder %s -size 100m %s"%( shellQuote(volname), shellQuote(os.path.join(WORKDIR, 'installer')), shellQuote(imagepath + ".tmp.dmg" ))) - + for i in range(5): + fd = os.popen(cmd, 'r') + data = fd.read() + xit = fd.close() + if not xit: + break + sys.stdout.write(data) + print(" -- retrying hdiutil create") + time.sleep(5) + else: + raise RuntimeError("command failed: %s"%(commandline,)) if not os.path.exists(os.path.join(WORKDIR, "mnt")): os.mkdir(os.path.join(WORKDIR, "mnt")) diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf index ebabb24..8133c35 100644 --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -1,5 +1,6 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf400 -{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPSMT;} +{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf100 +{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique; +\f3\fmodern\fcharset0 CourierNewPSMT;} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} \margl1440\margr1440\vieww15240\viewh15540\viewkind0 @@ -9,145 +10,143 @@ \ \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 2.7.15] -\b0 \ulnone \ -\ - -\b **NEW** -\b0 With Python 2.7.15, the python.org website now provides two installer variants for download: one that installs a -\i 64-bit-only -\i0 Python capable of running on -\i macOS 10.9 (Mavericks) -\i0 or later; and one that installs a -\i 64-bit/32-bit Intel -\i0 Python capable of running on -\i macOS 10.6 (Snow Leopard) -\i0 or later. (This ReadMe was installed with the -\i $MACOSX_DEPLOYMENT_TARGET -\i0 variant.) Previous Python 2.7.x releases provided the 10.6 or later installer and a 10.5 or later 32-bit-only variant. If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant. Use the 10.6 variant if you are running on macOS 10.6 through 10.8, if you need to maintain compatibility with previous 2.7.x releases, or if you want to produce standalone applications that can run on systems from 10.6. The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself. The list of these libraries varies by installer variant and is included at the end of the -\f1 License.rtf +\f1\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 2.7.15] +\f0\b0 \ulnone \ +\ +With Python 2.7.15, the python.org website now provides two installer variants for download: one that installs a +\f2\i 64-bit-only +\f0\i0 Python capable of running on +\f2\i macOS 10.9 (Mavericks) +\f0\i0 or later; and one that installs a +\f2\i 64-bit/32-bit Intel +\f0\i0 Python capable of running on +\f2\i macOS 10.6 (Snow Leopard) +\f0\i0 or later. (This ReadMe was installed with the +\f2\i $MACOSX_DEPLOYMENT_TARGET +\f0\i0 variant.) Previous Python 2.7.x releases provided the 10.6 or later installer and a 10.5 or later 32-bit-only variant. If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant. Use the 10.6 variant if you are running on macOS 10.6 through 10.8, if you need to maintain compatibility with previous 2.7.x releases, or if you want to produce standalone applications that can run on systems from 10.6. The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself. The list of these libraries varies by installer variant and is included at the end of the +\f3 License.rtf \f0 file.\ -\b \ul \ +\f1\b \ul \ Certificate verification and OpenSSL_[CHANGED in 2.7.15]\ -\b0 \ulnone \ +\f0\b0 \ulnone \ This variant of Python 2.7 now includes its own private copy of OpenSSL 1.0.2. Unlike previous releases, the deprecated Apple-supplied OpenSSL libraries are no longer used. This also means that the trust certificates in system and user keychains managed by the -\i Keychain Access -\i0 application and the -\i security -\i0 command line utility are no longer used as defaults by the Python -\f1 ssl +\f2\i Keychain Access +\f0\i0 application and the +\f2\i security +\f0\i0 command line utility are no longer used as defaults by the Python +\f3 ssl \f0 module. A sample command script is included in -\f1 /Applications/Python 2.7 +\f3 /Applications/Python 2.7 \f0 to install a curated bundle of default root certificates from the third-party -\f1 certifi +\f3 certifi \f0 package ({\field{\*\fldinst{HYPERLINK "https://pypi.python.org/pypi/certifi"}}{\fldrslt https://pypi.python.org/pypi/certifi}}). Click on -\f1 Install Certificates +\f3 Install Certificates \f0 to run it. If you choose to use -\f1 certifi +\f3 certifi \f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt project's email update service}} to be notified when the certificate bundle is updated.\ \ The bundled -\f1 pip +\f3 pip \f0 included with the Python 2.7 installer has its own default certificate store for verifying download connections.\ \ -\b \ul Using IDLE or other Tk applications [NEW/CHANGED in 2.7.15] -\b0 \ulnone \ +\f1\b \ul Using IDLE or other Tk applications [NEW/CHANGED in 2.7.15] +\f0\b0 \ulnone \ \ -The 10.9+ installer variant comes with its own private version of Tcl/Tk 8.6. It does not use system-supplied or third-party supplied versions of Tcl/Tk.\ +As of 2.7.15, the 10.9+ installer variant comes with its own private version of Tcl/Tk 8.6. It does not use system-supplied or third-party supplied versions of Tcl/Tk.\ \ -For the 10.6+ variant, you continue to need to install a newer third-party version of the -\i Tcl/Tk -\i0 8.5 (not 8.6) frameworks to use IDLE or other programs that use the Tkinter graphical user interface toolkit. Visit {\field{\*\fldinst{HYPERLINK "https://www.python.org/download/mac/tcltk/"}}{\fldrslt https://www.python.org/download/mac/tcltk/}} for current information about supported and recommended versions of -\i Tcl/Tk -\i0 for this version of Python and of macOS.\ +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 + +\f1\b \cf0 CHANGED in 2.7.16: +\f0\b0 The 10.6+ variant now also uses a private version of Tcl/Tk 8.6.\ +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 -\b \ul \ +\f1\b \cf0 \ul \ulc0 \ Binary installer support for Mac OS X 10.5 and earlier discontinued [CHANGED for Python 2.7.15] -\b0 \ulnone \ +\f0\b0 \ulnone \ \ As of Python 2.7.15, binary installers from python.org no longer support Mac OS X 10.5 (Leopard) systems. Binary installer support for Mac OS X 10.3.9 (Panther) and 10.4.x (Tiger) systems was previously dropped in Python 2.7.9. Mac OS X 10.5 was originally released by Apple in 2007 and last updated in 2009 and was the last OS X release for PPC machines (G4 and G5). If needed, it is still possible to build Python from source for 10.3.9, 10.4, or 10.5.\ \ -\b \ul Packages installed with the system Python 2.7 are no longer searched for [CHANGED for Python 2.7.13] -\b0 \ulnone \ +\f1\b \ul Packages installed with the system Python 2.7 are no longer searched for [CHANGED for Python 2.7.13] +\f0\b0 \ulnone \ \ As of Python 2.7.0, user-installed Python 2.7 versions from python.org installers added the system-wide site-packages directory for the Apple-supplied Python 2.7 to the end of their search path. This meant that packages installed with the system Python 2.7 could also be used by the user-installed Python 2.7. While sometimes convenient, this also often caused confusion with the implicit coupling between the two Python instances. Separately, as of macOS 10.12, Apple changed the layout of the system site-packages directory, -\f1 /Library/Python/2.7/site-packages +\f3 /Library/Python/2.7/site-packages \f0 , in a way that can now cause installation of -\f1 pip +\f3 pip \f0 components to fail. To avoid the confusion and the installation failures, as of 2.7.13 user-installed Pythons no longer add -\f1 /Library/Python/2.7/site-packages +\f3 /Library/Python/2.7/site-packages \f0 to -\f1 sys.path +\f3 sys.path \f0 . If you are using a package with both a user-installed Python 2.7 and the system Python 2.7, you will now need to ensure that separate copies of the package are installed for each instance.\ \ -\b \ul Installing on OS X 10.8 (Mountain Lion) or later systems [CHANGED for Python 2.7.9] -\b0 \ulnone \ +\f1\b \ul Installing on OS X 10.8 (Mountain Lion) or later systems [CHANGED for Python 2.7.9] +\f0\b0 \ulnone \ \ As of Python 2.7.9, installer packages from python.org are now compatible with the Gatekeeper security feature introduced in OS X 10.8. Downloaded packages can now be directly installed by double-clicking with the default system security settings. Python.org installer packages for macOS are signed with the Developer ID of the builder, as identified on {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt the download page}} for this release. To inspect the digital signature of the package, click on the lock icon in the upper right corner of the -\i Install Python -\i0 installer window. Refer to Apple\'92s support pages for {\field{\*\fldinst{HYPERLINK "http://support.apple.com/kb/ht5290"}}{\fldrslt more information on Gatekeeper}}.\ +\f2\i Install Python +\f0\i0 installer window. Refer to Apple\'92s support pages for {\field{\*\fldinst{HYPERLINK "http://support.apple.com/kb/ht5290"}}{\fldrslt more information on Gatekeeper}}.\ \ -\b \ul Simplified web-based installs [NEW for Python 2.7.9] -\b0 \ulnone \ +\f1\b \ul Simplified web-based installs [NEW for Python 2.7.9] +\f0\b0 \ulnone \ \ With the change to the newer flat format installer package, the download file now has a -\f1 .pkg +\f3 .pkg \f0 extension as it is no longer necessary to embed the installer within a disk image ( -\f1 .dmg +\f3 .dmg \f0 ) container. If you download the Python installer through a web browser, the macOS installer application may open automatically to allow you to perform the install. If your browser settings do not allow automatic open, double click on the downloaded installer file.\ \ -\b \ul New Installation Options and Defaults [NEW for Python 2.7.9] -\b0 \ulnone \ +\f1\b \ul New Installation Options and Defaults [NEW for Python 2.7.9] +\f0\b0 \ulnone \ \ The Python installer now includes an option to automatically install or upgrade -\f1 pip +\f3 pip \f0 , a tool for installing and managing Python packages. This option is enabled by default and no Internet access is required. If you do not want the installer to do this, select the -\i Customize -\i0 option at the -\i Installation Type -\i0 step and uncheck the -\i Install or upgrade pip -\i0 option. For other changes in this release, see the -\i Release Notes -\i0 link for this release at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\ +\f2\i Customize +\f0\i0 option at the +\f2\i Installation Type +\f0\i0 step and uncheck the +\f2\i Install or upgrade pip +\f0\i0 option. For other changes in this release, see the +\f2\i Release Notes +\f0\i0 link for this release at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\ \ -\b \ul Python 3 and Python 2 Co-existence\ +\f1\b \ul Python 3 and Python 2 Co-existence\ -\b0 \ulnone \ +\f0\b0 \ulnone \ Python.org Python 2.7 and 3.x versions can both be installed on your system and will not conflict. Python 2.7 command names contain a 2 or no digit: -\f1 python2 +\f3 python2 \f0 (or -\f1 python2.7 +\f3 python2.7 \f0 or -\f1 python +\f3 python \f0 ), -\f1 idle2 +\f3 idle2 \f0 (or -\f1 idle2.7 +\f3 idle2.7 \f0 or -\f1 idle +\f3 idle \f0 ), -\f1 pip2 +\f3 pip2 \f0 (or -\f1 pip2.7 +\f3 pip2.7 \f0 or -\f1 pip +\f3 pip \f0 ), etc. Command names for Python 3 contain a 3 in them, -\f1 python3 +\f3 python3 \f0 , -\f1 idle3 +\f3 idle3 \f0 , -\f1 pip3 +\f3 pip3 \f0 , etc. Also, installing a python.org Python 2.7 does not alter or remove any Apple-supplied system Pythons, found in -\f1 /usr/bin +\f3 /usr/bin \f0 .\ \ \ diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf index fcf9e04..4810c33 100644 --- a/Mac/BuildScript/resources/Welcome.rtf +++ b/Mac/BuildScript/resources/Welcome.rtf @@ -1,28 +1,33 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1561\cocoasubrtf400 -\cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPSMT;} +{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf100 +\cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT; +} {\colortbl;\red255\green255\blue255;} {\*\expandedcolortbl;;} \paperw11905\paperh16837\margl1440\margr1440\vieww13360\viewh10920\viewkind0 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 \f0\fs24 \cf0 This package will install -\b Python $FULL_VERSION -\b0 for -\b macOS $MACOSX_DEPLOYMENT_TARGET -\b0 .\ +\f1\b Python $FULL_VERSION +\f0\b0 for +\f1\b macOS $MACOSX_DEPLOYMENT_TARGET +\f0\b0 .\ \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0 \cf0 \ -\b Python for macOS -\b0 consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment -\b IDLE -\b0 .\ +\f1\b Python for macOS +\f0\b0 consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment +\f1\b IDLE +\f0\b0 .\ \ -\b NEW in 2.7.15: -\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support in the 10.9+ variant (no additional third-party downloads!), updated -\f1 pip, +\f1\b NEW in 2.7.15: +\f0\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support in the 10.9+ variant (no additional third-party downloads!), updated +\f2 pip, \f0 built-in OpenSSL 1.0.2 (click on -\f1 Install Certificates +\f2 Install Certificates \f0 for root certificates)\ +\ + +\f1\b CHANGED in 2.7.16: +\f0\b0 the 10.6+ variant now also uses a built-in Tcl/Tk 8.6\ }
\ No newline at end of file diff --git a/Mac/IDLE/Info.plist.in b/Mac/IDLE/Info.plist.in index 1d3c076..9f6beb6 100644 --- a/Mac/IDLE/Info.plist.in +++ b/Mac/IDLE/Info.plist.in @@ -36,7 +36,7 @@ <key>CFBundleExecutable</key> <string>IDLE</string> <key>CFBundleGetInfoString</key> - <string>%VERSION%, © 2001-2017 Python Software Foundation</string> + <string>%VERSION%, © 2001-2019 Python Software Foundation</string> <key>CFBundleIconFile</key> <string>IDLE.icns</string> <key>CFBundleIdentifier</key> diff --git a/Mac/PythonLauncher/Info.plist.in b/Mac/PythonLauncher/Info.plist.in index b84fffe..77dbb0f 100644 --- a/Mac/PythonLauncher/Info.plist.in +++ b/Mac/PythonLauncher/Info.plist.in @@ -40,7 +40,7 @@ <key>CFBundleExecutable</key> <string>PythonLauncher</string> <key>CFBundleGetInfoString</key> - <string>%VERSION%, © 2001-2018 Python Software Foundation</string> + <string>%VERSION%, © 2001-2019 Python Software Foundation</string> <key>CFBundleIconFile</key> <string>PythonLauncher.icns</string> <key>CFBundleIdentifier</key> diff --git a/Mac/Resources/app/Info.plist.in b/Mac/Resources/app/Info.plist.in index abe9ae2..b758198 100644 --- a/Mac/Resources/app/Info.plist.in +++ b/Mac/Resources/app/Info.plist.in @@ -37,7 +37,7 @@ <key>CFBundleInfoDictionaryVersion</key> <string>6.0</string> <key>CFBundleLongVersionString</key> - <string>%version%, (c) 2001-2018 Python Software Foundation.</string> + <string>%version%, (c) 2001-2019 Python Software Foundation.</string> <key>CFBundleName</key> <string>Python</string> <key>CFBundlePackageType</key> diff --git a/Mac/Resources/framework/Info.plist.in b/Mac/Resources/framework/Info.plist.in index c1ea9f6..0dc2e17 100644 --- a/Mac/Resources/framework/Info.plist.in +++ b/Mac/Resources/framework/Info.plist.in @@ -17,9 +17,9 @@ <key>CFBundlePackageType</key> <string>FMWK</string> <key>CFBundleShortVersionString</key> - <string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string> + <string>%VERSION%, (c) 2001-2019 Python Software Foundation.</string> <key>CFBundleLongVersionString</key> - <string>%VERSION%, (c) 2001-2018 Python Software Foundation.</string> + <string>%VERSION%, (c) 2001-2019 Python Software Foundation.</string> <key>CFBundleSignature</key> <string>????</string> <key>CFBundleVersion</key> @@ -94,6 +94,7 @@ Michael R Bax Anthony Baxter Mike Bayer Samuel L. Bayer +Bo Bayles Donald Beaudry David Beazley Carlo Beccarini @@ -138,6 +139,7 @@ Mike Bland Martin Bless Pablo Bleyer Erik van Blokland +Stéphane Blondon Eric Blossom Sergey Bobrov Finn Bock @@ -379,6 +381,7 @@ Ulrich Eckhardt David Edelsohn John Edmonds Grant Edwards +Zvi Effron John Ehresman Tal Einat Eric Eisner @@ -805,6 +808,7 @@ Ben Laurie Simon Law Julia Lawall Chris Lawrence +Michael Lazar Brian Leair Mathieu Leduc-Hamel Amandine Lee @@ -1291,6 +1295,7 @@ Roger D. Serwy Jerry Seutter Pete Sevander Denis Severson +Silas Sewell Ian Seyer Daniel Shahaf Mark Shannon @@ -1561,6 +1566,7 @@ Gordon Worley Darren Worrall Thomas Wouters Daniel Wozniak +Marcin Niemira Heiko Wundram Doug Wyatt Xiang Zhang @@ -1574,6 +1580,7 @@ Jason Yeo EungJun Yi Bob Yodlowski Danny Yoo +Wonsup Yoon Rory Yorke George Yoshida Kazuhiro Yoshida @@ -2,6 +2,409 @@ Python News +++++++++++ +What's New in Python 2.7.16 final? +================================== + +*Release date: 2019-03-02* + +IDLE +---- + +- bpo-32129: Avoid blurry IDLE application icon on macOS with Tk 8.6. Patch + by Kevin Walzer. + + +What's New in Python 2.7.16 release candidate 1? +================================================ + +*Release date: 2019-02-16* + +Security +-------- + +- bpo-35746: [CVE-2019-5010] Fix a NULL pointer deref in ssl module. The + cert parser did not handle CRL distribution points with empty DP or URI + correctly. A malicious or buggy certificate can result into segfault. + Vulnerability (TALOS-2018-0758) reported by Colin Read and Nicolas Edet of + Cisco. + +- bpo-16039: CVE-2013-1752: Change use of ``readline()`` in + :class:`imaplib.IMAP4_SSL` to limit line length. + +- bpo-28043: SSLContext has improved default settings: OP_NO_SSLv2, + OP_NO_SSLv3, OP_NO_COMPRESSION, OP_CIPHER_SERVER_PREFERENCE, + OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE and HIGH ciphers without MD5. + +- bpo-34791: The xml.sax and xml.dom.domreg no longer use environment + variables to override parser implementations when + sys.flags.ignore_environment is set by -E or -I arguments. + +- bpo-34623: CVE-2018-14647: The C accelerated _elementtree module now + initializes hash randomization salt from _Py_HashSecret instead of + libexpat's default CSPRNG. + +- bpo-34540: When ``shutil.make_archive`` falls back to the external ``zip`` + problem, it uses :mod:`subprocess` to invoke it rather than + :mod:`distutils.spawn`. This closes a possible shell injection vector. + +- bpo-34405: Updated to OpenSSL 1.0.2p for Windows builds. + +Core and Builtins +----------------- + +- bpo-35552: Format character ``%s`` in :c:func:`PyString_FromFormat` no + longer read memory past the limit if *precision* is specified. + +- bpo-35504: Fix segfaults and :exc:`SystemError`\ s when deleting certain + attributes. Patch by Zackery Spytz. + +- bpo-35368: :c:func:`PyMem_Malloc` is now also thread-safe in debug mode. + +- bpo-35214: Fixed an out of bounds memory access when parsing a truncated + unicode escape sequence at the end of a string such as ``u'\N'``. It + would read one byte beyond the end of the memory allocation. + +- bpo-34974: The :class:`bytearray` constructor no longer convert unexpected + exceptions (e.g. :exc:`MemoryError` and :exc:`KeyboardInterrupt`) to + :exc:`TypeError`. + +- bpo-22851: Fix a segfault when accessing + ``generator.gi_frame.f_restricted`` when the generator is exhausted. + Patch by Zackery Spytz. + +- bpo-18560: Fix potential NULL pointer dereference in sum(). + +- bpo-34400: Fix undefined behavior in parsetok.c. Patch by Zackery Spytz. + +- bpo-25083: Adding I/O error checking when reading .py files and aborting + importing on error. + +- bpo-25943: Fix potential heap corruption in the :mod:`bsddb` module. + Patch by Zackery Spytz. + +- bpo-34068: In :meth:`io.IOBase.close`, ensure that the + :attr:`~io.IOBase.closed` attribute is not set with a live exception. + Patch by Zackery Spytz and Serhiy Storchaka. + +- bpo-34080: Fixed a memory leak in the compiler when it raised some + uncommon errors during tokenizing. + +- bpo-33956: Update vendored Expat library copy to version 2.2.5. + +- bpo-30654: Fixed reset of the SIGINT handler to SIG_DFL on interpreter + shutdown even when there was a custom handler set previously. Patch by + Philipp Kerling. + +- bpo-33645: Fixed an "unknown parsing error" on parsing the "<>" operator + when run Python with both options ``-3`` and ``-We``. + +- bpo-33622: Fixed a leak when the garbage collector fails to add an object + with the ``__del__`` method or referenced by it into the + :data:`gc.garbage` list. :c:func:`PyGC_Collect` can now be called when an + exception is set and preserves it. + +- bpo-33391: Fix a leak in set_symmetric_difference(). + +- bpo-25750: Fix rare Python crash due to bad refcounting in + ``type_getattro()`` if a descriptor deletes itself from the class. Patch + by Jeroen Demeyer. + +- bpo-25862: Fix assertion failures in the ``tell()`` method of + ``io.TextIOWrapper``. Patch by Zackery Spytz. + +Library +------- + +- bpo-8765: The write() method of buffered and unbuffered binary streams in + the io module emits now a DeprecationWarning in Py3k mode for unicode + argument. + +- bpo-35052: Fix xml.dom.minidom cloneNode() on a document with an entity: + pass the correct arguments to the user data handler of an entity. + +- bpo-10496: :func:`~distutils.utils.check_environ` of + :mod:`distutils.utils` now catchs :exc:`KeyError` on calling + :func:`pwd.getpwuid`: don't create the ``HOME`` environment variable in + this case. + +- bpo-10496: :func:`posixpath.expanduser` now returns the input *path* + unchanged if the ``HOME`` environment variable is not set and the current + user has no home directory (if the current user identifier doesn't exist + in the password database). This change fix the :mod:`site` module if the + current user doesn't exist in the password database (if the user has no + home directory). + +- bpo-24746: Avoid stripping trailing whitespace in doctest fancy diff. + Orignial patch by R. David Murray & Jairo Trad. Enhanced by Sanyam + Khurana. + +- bpo-35277: Update ensurepip to install pip 18.1 and setuptools 40.6.2. + +- bpo-35062: Fix incorrect parsing of + :class:`_io.IncrementalNewlineDecoder`'s *translate* argument. + +- bpo-35079: Improve difflib.SequenceManager.get_matching_blocks doc by + adding 'non- overlapping' and changing '!=' to '<'. + +- bpo-35017: :meth:`socketserver.BaseServer.serve_forever` now exits + immediately if it's :meth:`~socketserver.BaseServer.shutdown` method is + called while it is polling for new events. + +- bpo-34794: Fixed a leak in Tkinter when pass the Python wrapper around + Tcl_Obj back to Tcl/Tk. + +- bpo-23420: Verify the value for the parameter '-s' of the cProfile CLI. + Patch by Robert Kuska + +- bpo-16965: The :term:`2to3` :2to3fixer:`execfile` fixer now opens the file + with mode ``'rb'``. Patch by Zackery Spytz. + +- bpo-34936: Fix ``TclError`` in ``tkinter.Spinbox.selection_element()``. + Patch by Juliette Monsel. + +- bpo-34866: Adding ``max_num_fields`` to ``cgi.FieldStorage`` to make DOS + attacks harder by limiting the number of ``MiniFieldStorage`` objects + created by ``FieldStorage``. + +- bpo-34738: ZIP files created by :mod:`distutils` will now include entries + for directories. + +- bpo-34652: Ensure :func:`os.lchmod` is never defined on Linux. + +- bpo-34625: Update vendorized expat library version to 2.2.6. + +- bpo-34610: Fixed iterator of :class:`multiprocessing.managers.DictProxy`. + +- bpo-34530: ``distutils.spawn.find_executable()`` now falls back on + :data:`os.defpath` if the ``PATH`` environment variable is not set. + +- bpo-34472: Improved compatibility for streamed files in :mod:`zipfile`. + Previously an optional signature was not being written and certain ZIP + applications were not supported. Patch by Silas Sewell. + +- bpo-6700: Fix inspect.getsourcelines for module level frames/tracebacks. + Patch by Vladimir Matveev. + +- bpo-31715: Associate ``.mjs`` file extension with + ``application/javascript`` MIME Type. + +- bpo-32947: Add OP_ENABLE_MIDDLEBOX_COMPAT and test workaround for TLSv1.3 + for future compatibility with OpenSSL 1.1.1. + +- bpo-34341: Appending to the ZIP archive with the ZIP64 extension no longer + grows the size of extra fields of existing entries. + +- bpo-34052: :meth:`sqlite3.Connection.create_aggregate`, + :meth:`sqlite3.Connection.create_function`, + :meth:`sqlite3.Connection.set_authorizer`, + :meth:`sqlite3.Connection.set_progress_handler` methods raises TypeError + when unhashable objects are passed as callable. These methods now don't + pass such objects to SQLite API. Previous behavior could lead to + segfaults. Patch by Sergey Fedoseev. + +- bpo-34019: webbrowser: Correct the arguments passed to Opera Browser when + opening a new URL using the ``webbrowser`` module. Patch by Bumsik Kim. + +- bpo-33974: Fixed passing lists and tuples of strings containing special + characters ``"``, ``\``, ``{``, ``}`` and ``\n`` as options to + :mod:`~tkinter.ttk` widgets. + +- bpo-26544: Fixed implementation of :func:`platform.libc_ver`. It almost + always returned version '2.9' for glibc. + +- bpo-33767: The concatenation (``+``) and repetition (``*``) sequence + operations now raise :exc:`TypeError` instead of :exc:`SystemError` when + performed on :class:`mmap.mmap` objects. Patch by Zackery Spytz. + +- bpo-11874: Use a better regex when breaking usage into wrappable parts. + Avoids bogus assertion errors from custom metavar strings. + +- bpo-33570: Change TLS 1.3 cipher suite settings for compatibility with + OpenSSL 1.1.1-pre6 and newer. OpenSSL 1.1.1 will have TLS 1.3 cipers + enabled by default. + +- bpo-33542: Prevent ``uuid.get_node`` from using a DUID instead of a MAC on + Windows. Patch by Zvi Effron + +- bpo-33096: Removed unintentionally backported from Python 3 Tkinter files. + +- bpo-20087: Updated alias mapping with glibc 2.27 supported locales. + +- bpo-33422: Fix trailing quotation marks getting deleted when looking up + byte/string literals on pydoc. Patch by Andrés Delfino. + +- bpo-33336: ``imaplib`` now allows ``MOVE`` command in ``IMAP4.uid()`` (RFC + 6851: IMAP MOVE Extension) and potentially as a name of supported method + of ``IMAP4`` object. + +- bpo-33359: Fix running ``python -m curses.has_key``. + +- bpo-33131: Upgrade bundled version of pip to 10.0.1. + +- bpo-33308: Fixed a crash in the :mod:`parser` module when converting an ST + object to a tree of tuples or lists with ``line_info=False`` and + ``col_info=True``. + +- bpo-33256: Fix display of ``<module>`` call in the html produced by + ``cgitb.html()``. Patch by Stéphane Blondon. + +- bpo-32861: The urllib.robotparser's ``__str__`` representation now + includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. + Patch by Michael Lazar. + +- bpo-33038: gzip.GzipFile no longer produces an AttributeError exception + when used with a file object with a non-string name attribute. Patch by Bo + Bayles. + +- bpo-32857: In :mod:`tkinter`, ``after_cancel(None)`` now raises a + :exc:`ValueError` instead of canceling the first scheduled function. + Patch by Cheryl Sabella. + +- bpo-32502: uuid.uuid1 no longer raises an exception if a 64-bit hardware + address is encountered. + +- bpo-31608: Raise a ``TypeError`` instead of crashing if a + ``collections.deque`` subclass returns a non-deque from ``__new__``. Patch + by Oren Milman. + +- bpo-16865: Support arrays >=2GiB in :mod:`ctypes`. Patch by Segev Finer. + +- bpo-29456: Fix bugs in hangul normalization: u1176, u11a7 and u11c3 + +Documentation +------------- + +- bpo-35035: Rename documentation for :mod:`email.utils` to + ``email.utils.rst``. + +- bpo-34967: Use app.add_object_type() instead of the deprecated Sphinx + function app.description_unit() + +- bpo-13407: Add a note to :mod:`bz2` and :mod:`tarfile` stating that + handling of multi-stream bzip2 files is not supported. + +- bpo-33503: Fix broken pypi link + +Tests +----- + +- bpo-34279: :func:`test.support.run_unittest` no longer raise + :exc:`TestDidNotRun` if the test result contains skipped tests. The + exception is now only raised if no test have been run and no test have + been skipped. + +- bpo-34279: regrtest issue a warning when no tests have been executed in a + particular test file. Also, a new final result state is issued if no test + have been executed across all test files. Patch by Pablo Galindo. + +- bpo-34661: Fix test_shutil if unzip doesn't support -t. + +- bpo-34542: Use 3072 RSA keys and SHA-256 signature for test certs and + keys. + +- bpo-34391: Fix ftplib test for TLS 1.3 by reading from data socket. + +- bpo-34399: Update all RSA keys and DH params to use at least 2048 bits. + +- bpo-33901: Fix test_gdbm on macOS with gdbm 1.15: add a larger value to + make sure that the file size changes. + +- bpo-33873: Fix a bug in ``regrtest`` that caused an extra test to run if + --huntrleaks/-R was used. Exit with error in case that invalid parameters + are specified to --huntrleaks/-R (at least one warmup run and one + repetition must be used). + +- bpo-29512: Rename Lib/test/bisect.py to Lib/test/bisect_cmd.py. The old + name was in conflict with Lib/bisect.py, causing test failures, depending + how tests were run. + +- bpo-32962: Fixed test_gdb when Python is compiled with flags -mcet + -fcf-protection -O0. + +- bpo-33354: Skip ``test_ssl.test_load_dh_params`` when Python filesystem + encoding cannot encode the provided path. + +- bpo-19417: Add test_bdb.py. + +Build +----- + +- bpo-35139: Fix a compiler error when statically linking `pyexpat` in + `Modules/Setup`. + +- bpo-34710: Fixed SSL module build with OpenSSL & pedantic CFLAGS. + +- bpo-33015: Fix an undefined behaviour in the pthread implementation of + :c:func:`PyThread_start_new_thread`: add a function wrapper to always + return ``NULL``. + +- bpo-30345: Add -g to LDFLAGS when compiling with LTO to get debug symbols. + +Windows +------- + +- bpo-35401: Updates Windows build to OpenSSL 1.0.2q + +- bpo-34603: Fix returning structs from functions produced by MSVC + +- bpo-33711: Fixed licence generation error when building the installer. + +macOS +----- + +- bpo-15663: The macOS 10.6+ installer now provides a private copy of Tcl/Tk + 8.6, like the 10.9+ installer does. + +- bpo-35401: Update macOS installer to use OpenSSL 1.0.2q. + +- bpo-34405: Update to OpenSSL 1.0.2p for macOS installer builds. + +IDLE +---- + +- bpo-34275: Make IDLE calltips always visible on Mac. Some MacOS-tk + combinations need .update_idletasks(). Patch by Kevin Walzer. + +- bpo-34120: Fix unresponsiveness after closing certain windows and dialogs. + +- bpo-33856: Add "help" in the welcome message of IDLE + +- bpo-31500: Default fonts now are scaled on HiDPI displays. + +Tools/Demos +----------- + +- bpo-34989: python-gdb.py now handles errors on computing the line number + of a Python frame. + +- bpo-34500: Fix 2 ResourceWarning in difflib.py. Patch by Mickaël + Schoentgen. + +- bpo-29367: python-gdb.py now supports also method-wrapper (wrapperobject) + objects. + +- bpo-32962: python-gdb now catchs ``UnicodeDecodeError`` exceptions when + calling ``string()``. + +- bpo-32962: python-gdb now catchs ValueError on read_var(): when Python has + no debug symbols for example. + +C API +----- + +- bpo-33817: Fixed :c:func:`_PyString_Resize` and + :c:func:`_PyUnicode_Resize` for empty strings. This fixed also + :c:func:`PyString_FromFormat` and :c:func:`PyUnicode_FromFormat` when they + return an empty string (e.g. ``PyString_FromFormat("%s", "")``). + +- bpo-34229: Check start and stop of slice object to be long when they are + not int in :c:func:`PySlice_GetIndices`. + +- bpo-23927: Fixed :exc:`SystemError` in + :c:func:`PyArg_ParseTupleAndKeywords` when the ``w*`` format unit is used + for optional parameter. + + What's New in Python 2.7.15 final? ================================== @@ -47,7 +450,7 @@ Security infinite loop DoS), CVE-2016-9063 (Integer overflow, re-fix), CVE-2016-0718 (Fix regression bugs from 2.2.0's fix to CVE-2016-0718) and CVE-2012-0876 (Counter hash flooding with SipHash). Note: the - CVE-2016-5300 (Use os- specific entropy sources like getrandom) doesn't + CVE-2016-5300 (Use os-specific entropy sources like getrandom) doesn't impact Python, since Python already gets entropy from the OS to set the expat secret using ``XML_SetHashSalt()``. @@ -124,8 +527,8 @@ Core and Builtins mutated during searching, inserting or comparing. Based on patches by Duane Griffin and Tim Mitchell. -- bpo-25794: Fixed type.__setattr__() and type.__delattr__() for non- - interned or unicode attribute names. Based on patch by Eryk Sun. +- bpo-25794: Fixed type.__setattr__() and type.__delattr__() for + non-interned or unicode attribute names. Based on patch by Eryk Sun. - bpo-29935: Fixed error messages in the index() method of tuple and list when pass indices of wrong type. @@ -291,8 +694,8 @@ Library - bpo-29136: Add TLS 1.3 cipher suites and OP_NO_TLSv1_3. - bpo-31334: Fix ``poll.poll([timeout])`` in the ``select`` module for - arbitrary negative timeouts on all OSes where it can only be a non- - negative integer or -1. Patch by Riccardo Coccioli. + arbitrary negative timeouts on all OSes where it can only be a + non-negative integer or -1. Patch by Riccardo Coccioli. - bpo-10746: Fix ctypes producing wrong PEP 3118 type codes for integer types. @@ -313,9 +716,6 @@ Library - bpo-30746: Prohibited the '=' character in environment variable names in ``os.putenv()`` and ``os.spawn*()``. -- bpo-28994: The traceback no longer displayed for SystemExit raised in a - callback registered by atexit. - - bpo-30418: On Windows, subprocess.Popen.communicate() now also ignore EINVAL on stdin.write() if the child process is still running but closed the pipe. @@ -470,7 +870,7 @@ Documentation - bpo-12067: Rewrite Comparisons section in the Expressions chapter of the language reference. Some of the details of comparing mixed types were incorrect or ambiguous. Added default behaviour and consistency - suggestions for user- defined classes. Based on patch from Andy Maier. + suggestions for user-defined classes. Based on patch from Andy Maier. Tests ----- @@ -532,7 +932,7 @@ Build - bpo-29243: Prevent unnecessary rebuilding of Python during ``make test``, ``make install`` and some other make targets when configured with - ``--enable- optimizations``. + ``--enable-optimizations``. - bpo-23404: Don't regenerate generated files based on file modification time anymore: the action is now explicit. Replace ``make touch`` with @@ -648,7 +1048,7 @@ Library contains CR or LF. Patch by Dong-hee Na. - bpo-30595: multiprocessing.Queue.get() with a timeout now polls its reader - in non- blocking mode if it succeeded to aquire the lock but the acquire + in non-blocking mode if it succeeded to aquire the lock but the acquire took longer than the timeout. - bpo-29902: Py3k deprecation warning now is emitted when pickling or @@ -725,7 +1125,7 @@ Core and Builtins tp_free (e.g. C-inherited Cython classes). - bpo-19398: Extra slash no longer added to sys.path components in case of - empty compile- time PYTHONPATH components. + empty compile-time PYTHONPATH components. - bpo-21720: Improve exception message when the type of fromlist is unicode. fromlist parameter of __import__() only accepts str in Python 2 and this @@ -836,7 +1236,7 @@ Library - bpo-24363: When parsing HTTP header fields, if an invalid line is encountered, skip it and continue parsing. Previously, no more header fields were parsed, which could lead to fields for HTTP framing like - Content-Length and Transfer- Encoding being overlooked. + Content-Length and Transfer-Encoding being overlooked. - bpo-27599: Fixed buffer overrun in binascii.b2a_qp() and binascii.a2b_qp(). @@ -1014,8 +1414,8 @@ Build - bpo-27983: Cause lack of llvm-profdata tool when using clang as required for PGO linking to be a configure time error rather than make time when - --with- optimizations is enabled. Also improve our ability to find the - llvm- profdata tool on MacOS and some Linuxes. + ``--with-optimizations`` is enabled. Also improve our ability to find the + llvm-profdata tool on MacOS and some Linuxes. - bpo-26359: Add the --with-optimizations configure flag. @@ -1301,8 +1701,8 @@ Library - bpo-19883: Fixed possible integer overflows in zipimport. -- bpo-26147: xmlrpclib now works with unicode not encodable with used non- - UTF-8 encoding. +- bpo-26147: xmlrpclib now works with unicode not encodable with used + non-UTF-8 encoding. - bpo-16620: Fixed AttributeError in msilib.Directory.glob(). @@ -1455,9 +1855,9 @@ Tools/Demos ----------- - bpo-26799: Fix python-gdb.py: don't get C types once when the Python code - is loaded, but get C types on demand. The C types can change if python- - gdb.py is loaded before the Python executable. Patch written by Thomas - Ilsche. + is loaded, but get C types on demand. The C types can change if + python-gdb.py is loaded before the Python executable. Patch written by + Thomas Ilsche. C API ----- @@ -1659,7 +2059,7 @@ Library whitespace. - bpo-20387: Restore semantic round-trip correctness in tokenize/untokenize - for tab- indented blocks. + for tab-indented blocks. - bpo-24456: Fixed possible buffer over-read in adpcm2lin() and lin2adpcm() functions of the audioop module. Fixed SystemError when the state is not @@ -1780,10 +2180,10 @@ IDLE - bpo-25198: Enhance the initial html viewer now used for Idle Help. * Properly indent fixed-pitch text (patch by Mark Roseman). * Give code - snippet a very Sphinx- like light blueish-gray background. * Re-use - initial width and height set by users for shell and editor. * When the - Table of Contents (TOC) menu is used, put the section header at the top of - the screen. + snippet a very Sphinx-like light blueish-gray background. * Re-use initial + width and height set by users for shell and editor. * When the Table of + Contents (TOC) menu is used, put the section header at the top of the + screen. - bpo-25225: Condense and rewrite Idle doc section on text colors. @@ -1815,10 +2215,11 @@ IDLE - bpo-23672: Allow Idle to edit and run files with astral chars in name. Patch by Mohd Sanad Zaki Rizvi. -- bpo-24745: Idle editor default font. Switch from Courier to platform- - sensitive TkFixedFont. This should not affect current customized font - selections. If there is a problem, edit $HOME/.idlerc/config-main.cfg and - remove 'fontxxx' entries from [Editor Window]. Patch by Mark Roseman. +- bpo-24745: Idle editor default font. Switch from Courier to + platform-sensitive TkFixedFont. This should not affect current customized + font selections. If there is a problem, edit + $HOME/.idlerc/config-main.cfg and remove 'fontxxx' entries from [Editor + Window]. Patch by Mark Roseman. - bpo-21192: Idle editor. When a file is run, put its name in the restart bar. Do not print false prompts. Original patch by Adnan Umer. @@ -1845,8 +2246,8 @@ Tests ----- - bpo-24751: When running regrtest with the ``-w`` command line option, a - test run is no longer marked as a failure if all tests succeed when re- - run. + test run is no longer marked as a failure if all tests succeed when + re-run. - PCbuild\rt.bat now accepts an unlimited number of arguments to pass along to regrtest.py. Previously there was a limit of 9. @@ -2036,8 +2437,8 @@ Library the multiprocessing module. - bpo-21840: Fixed expanding unicode variables of form $var in - posixpath.expandvars(). Fixed all os.path implementations on unicode- - disabled builds. + posixpath.expandvars(). Fixed all os.path implementations on + unicode-disabled builds. - bpo-23367: Fix possible overflows in the unicodedata module. @@ -2565,7 +2966,7 @@ Build now display special message when and only when there are failures. - bpo-21166: Prevent possible segfaults and other random failures of python - --generate- posix-vars in pybuilddir.txt build target. + ``--generate-posix-vars`` in pybuilddir.txt build target. - bpo-18096: Fix library order returned by python-config. @@ -2606,8 +3007,8 @@ Core and Builtins - bpo-21831: Avoid integer overflow when large sizes and offsets are given to the buffer type. CVE-2014-7185. -- bpo-19656: Running Python with the -3 option now also warns about non- - ascii bytes literals. +- bpo-19656: Running Python with the -3 option now also warns about + non-ascii bytes literals. - bpo-21642: If the conditional if-else expression, allow an integer written with no space between itself and the ``else`` keyword (e.g. ``True if @@ -3034,8 +3435,8 @@ Library compiled on a system with SO_REUSEPORT defined in the headers but run on a system with an OS kernel that does not support that new socket option. -- bpo-19633: Fixed writing not compressed 16- and 32-bit wave files on big- - endian platforms. +- bpo-19633: Fixed writing not compressed 16- and 32-bit wave files on + big-endian platforms. - bpo-19449: in csv's writerow, handle non-string keys when generating the error message that certain keys are not in the 'fieldnames' list. @@ -3118,8 +3519,8 @@ IDLE - bpo-18270: Prevent possible IDLE AttributeError on OS X when no initial shell window is present. -- bpo-17654: Ensure IDLE menus are customized properly on OS X for non- - framework builds and for all variants of Tk. +- bpo-17654: Ensure IDLE menus are customized properly on OS X for + non-framework builds and for all variants of Tk. Tests ----- @@ -3711,8 +4112,8 @@ Library - bpo-17016: Get rid of possible pointer wraparounds and integer overflows in the re module. Patch by Nickolai Zeldovich. -- bpo-17536: Add to webbrowser's browser list: xdg-open, gvfs-open, www- - browser, x-www- browser, chromium browsers, iceweasel, iceape. +- bpo-17536: Add to webbrowser's browser list: xdg-open, gvfs-open, + www-browser, x-www-browser, chromium browsers, iceweasel, iceape. - bpo-17656: Fix extraction of zip files with unicode member paths. @@ -3961,8 +4362,8 @@ Core and Builtins - bpo-14579: Fix error handling bug in the utf-16 decoder. Patch by Serhiy Storchaka. -- bpo-15368: An issue that caused bytecode generation to be non- - deterministic when using randomized hashing (-R) has been fixed. +- bpo-15368: An issue that caused bytecode generation to be + non-deterministic when using randomized hashing (-R) has been fixed. - bpo-15897: zipimport.c doesn't check return value of fseek(). Patch by Felipe Cruz. @@ -4092,8 +4493,8 @@ Library - bpo-13153: Tkinter functions now raise TclError instead of ValueError when a unicode argument contains non-BMP character. -- bpo-9669: Protect re against infinite loops on zero-width matching in non- - greedy repeat. Patch by Matthew Barnett. +- bpo-9669: Protect re against infinite loops on zero-width matching in + non-greedy repeat. Patch by Matthew Barnett. - bpo-13169: The maximal repetition number in a regular expression has been increased from 65534 to 2147483647 (on 32-bit platform) or 4294967294 (on @@ -4277,7 +4678,7 @@ Library expected in subprocess.Popen.wait(). - bpo-16411: Fix a bug where zlib.decompressobj().flush() might try to - access previously- freed memory. Patch by Serhiy Storchaka. + access previously-freed memory. Patch by Serhiy Storchaka. - bpo-16350: zlib.decompressobj().decompress() now accumulates data from successive calls after EOF in unused_data, instead of only saving the @@ -4364,7 +4765,7 @@ Library Patch by Bohuslav Kabrda. - bpo-15477: In cmath and math modules, add workaround for platforms whose - system- supplied log1p function doesn't respect signs of zeros. + system-supplied log1p function doesn't respect signs of zeros. - bpo-11062: Fix adding a message from file to Babyl mailbox. @@ -4605,16 +5006,16 @@ Library - bpo-13590: OS X Xcode 4 - improve support for universal extension modules In particular, fix extension module build failures when trying to use - 32-bit- only installer Pythons on systems with Xcode 4 (currently OS X + 32-bit-only installer Pythons on systems with Xcode 4 (currently OS X 10.8, 10.7, and optionally 10.6). * Backport 3.3.0 fixes to 2.7 branch (for release in 2.7.4) * Since Xcode 4 removes ppc support, extension module builds now check for ppc compiler support and by default remove ppc - and ppc64 archs when they are not available. * Extension module builds - now revert to using system installed headers and libs (/usr and + and ppc64 archs when they are not available. * Extension module builds now + revert to using system installed headers and libs (/usr and /System/Library) if the SDK used to build the interpreter is not installed or has moved. * Try to avoid building extension modules with - deprecated and problematic Apple llvm-gcc compiler. If original - compiler is not available, use clang instead by default. + deprecated and problematic Apple llvm-gcc compiler. If original compiler + is not available, use clang instead by default. IDLE ---- @@ -4706,8 +5107,8 @@ Tests test_email regression test. test_email_renamed contains some tests that test_email does not. -- bpo-17041: Fix testing when Python is configured with the --without-doc- - strings option. +- bpo-17041: Fix testing when Python is configured with the + --without-doc-strings option. - bpo-15539: Added regression tests for Tools/scripts/pindent.py. @@ -4992,8 +5393,8 @@ Core and Builtins - bpo-12610: Verify that user generated AST has correct string and identifier types before compiling. (See also: bpo-12609) -- bpo-11627: Fix segfault when __new__ on an exception returns a non- - exception class. +- bpo-11627: Fix segfault when __new__ on an exception returns a + non-exception class. - bpo-12149: Update the method cache after a type's dictionnary gets cleared by the garbage collector. This fixes a segfault when an instance and its @@ -5214,9 +5615,9 @@ Library - bpo-12931: xmlrpclib now encodes Unicode URI to ISO-8859-1, instead of failing with a UnicodeDecodeError. -- bpo-8933: distutils' PKG-INFO files will now correctly report Metadata- - Version: 1.1 instead of 1.0 if a Classifier or Download-URL field is - present. +- bpo-8933: distutils' PKG-INFO files will now correctly report + Metadata-Version: 1.1 instead of 1.0 if a Classifier or Download-URL field + is present. - bpo-8286: The distutils command sdist will print a warning message instead of crashing when an invalid path is given in the manifest template. @@ -5265,8 +5666,8 @@ Library - bpo-11933: Fix incorrect mtime comparison in distutils. -- bpo-11104: Fix the behavior of distutils' sdist command with manually- - maintained MANIFEST files. (See also: bpo-8688) +- bpo-11104: Fix the behavior of distutils' sdist command with + manually-maintained MANIFEST files. (See also: bpo-8688) - bpo-8887: "pydoc somebuiltin.somemethod" (or help('somebuiltin.somemethod') in Python code) now finds the doc of the @@ -5380,8 +5781,8 @@ Library - bpo-9975: socket: Fix incorrect use of flowinfo and scope_id. Patch by Vilmos Nebehaj. -- bpo-13159: FileIO, BZ2File, and the built-in file class now use a linear- - time buffer growth strategy instead of a quadratic one. +- bpo-13159: FileIO, BZ2File, and the built-in file class now use a + linear-time buffer growth strategy instead of a quadratic one. - bpo-13070: Fix a crash when a TextIOWrapper caught in a reference cycle would be finalized after the reference to its underlying BufferedRWPair's @@ -5582,7 +5983,7 @@ Core and Builtins to be propagated. - bpo-1195: Fix input() if it is interrupted by CTRL+d and then CTRL+c, - clear the end- of-file indicator after CTRL+d. + clear the end-of-file indicator after CTRL+d. - bpo-8651: PyArg_Parse*() functions raise an OverflowError if the file doesn't have PY_SSIZE_T_CLEAN define and the size doesn't fit in an int @@ -5624,9 +6025,9 @@ Core and Builtins - bpo-10892: Don't segfault when trying to delete __abstractmethods__ from a class. -- bpo-8020: Avoid a crash where the small objects allocator would read non- - Python managed memory while it is being modified by another thread. Patch - by Matt Bandy. +- bpo-8020: Avoid a crash where the small objects allocator would read + non-Python managed memory while it is being modified by another thread. + Patch by Matt Bandy. - bpo-11004: Repaired edge case in deque.count(). @@ -5830,14 +6231,14 @@ Library 32-bit Windows. - bpo-10360: In WeakSet, do not raise TypeErrors when testing for membership - of non- weakrefable objects. + of non-weakrefable objects. - bpo-10549: Fix pydoc traceback when text-documenting certain classes. - bpo-940286: pydoc.Helper.help() ignores input/output init parameters. -- bpo-11171: Fix detection of config/Makefile when --prefix != --exec- - prefix, which caused Python to not start. +- bpo-11171: Fix detection of config/Makefile when --prefix != + --exec-prefix, which caused Python to not start. - bpo-11116: any error during addition of a message to a mailbox now causes a rollback, instead of leaving the mailbox partially modified. @@ -5854,7 +6255,7 @@ Library symbol named FSTAT. - bpo-10916: mmap should not segfault when a file is mapped using 0 as - length and a non- zero offset, and an attempt to read past the end of file + length and a non-zero offset, and an attempt to read past the end of file is made (IndexError is raised instead). Patch by Ross Lagerwall. - bpo-10875: Update Regular Expression HOWTO; patch by 'SilentGhost'. @@ -6036,7 +6437,7 @@ Build installers now includes a link to the installed documentation. - bpo-11054: Allow Mac OS X installer builds to again work on 10.5 with the - system- provided Python. + system-provided Python. - bpo-10843: Update third-party library versions used in OS X 32-bit installer builds: bzip2 1.0.6, readline 6.1.2, SQLite 3.7.4 (with @@ -6205,8 +6606,8 @@ Core and Builtins - bpo-10006: type.__abstractmethods__ now raises an AttributeError. -- bpo-9797: pystate.c wrongly assumed that zero couldn't be a valid thread- - local storage key. +- bpo-9797: pystate.c wrongly assumed that zero couldn't be a valid + thread-local storage key. - bpo-4947: The write() method of sys.stdout and sys.stderr uses their encoding and errors attributes instead of using utf-8 in strict mode, to @@ -6241,8 +6642,8 @@ Library - bpo-4471: Properly shutdown socket in IMAP.shutdown(). Patch by Lorenzo M. Catucci. -- bpo-10126: Fix distutils' test_build when Python was built with --enable- - shared. +- bpo-10126: Fix distutils' test_build when Python was built with + --enable-shared. - Fix typo in one sdist option (medata-check). @@ -6493,8 +6894,8 @@ Library - bpo-8015: In pdb, do not crash when an empty line is entered as a breakpoint command. -- bpo-9448: Fix a leak of OS resources (mutexes or semaphores) when re- - initializing a buffered IO object by calling its ``__init__`` method. +- bpo-9448: Fix a leak of OS resources (mutexes or semaphores) when + re-initializing a buffered IO object by calling its ``__init__`` method. - bpo-7909: Do not touch paths with the special prefixes ``\\.\`` or ``\\?\`` in ntpath.normpath(). @@ -6849,8 +7250,8 @@ C API - bpo-5753: A new C API function, :cfunc:`PySys_SetArgvEx`, allows embedders of the interpreter to set sys.argv without also modifying sys.path. This - helps fix `CVE-2008-5983 <http://cve.mitre.org/cgi- - bin/cvename.cgi?name=CVE-2008-5983>`_. + helps fix `CVE-2008-5983 + <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_. Library ------- @@ -6935,8 +7336,8 @@ Tests - bpo-7449: Skip test_socketserver if threading support is disabled - On darwin, ``test_site`` assumed that a framework build was being used, - leading to a failure where four directories were expected for site- - packages instead of two in a non-framework build. + leading to a failure where four directories were expected for + site-packages instead of two in a non-framework build. Build ----- @@ -6964,7 +7365,7 @@ Core and Builtins - Run Clang 2.7's static analyzer for ``Objects/`` and ``Python/``. - bpo-1533: Fix inconsistency in range function argument processing: any - non-float non- integer argument is now converted to an integer (if + non-float non-integer argument is now converted to an integer (if possible) using its __int__ method. Previously, only small arguments were treated this way; larger arguments (those whose __int__ was outside the range of a C long) would produce a TypeError. @@ -7209,7 +7610,7 @@ Core and Builtins - Raise a TypeError when trying to delete a T_STRING_INPLACE struct member. - bpo-7994: Issue a PendingDeprecationWarning if object.__format__ is called - with a non- empty format string. This is an effort to future-proof user + with a non-empty format string. This is an effort to future-proof user code. If a derived class does not currently implement __format__ but later adds its own __format__, it would most likely break user code that had supplied a format string. This will be changed to a DeprecationWarning in @@ -7257,8 +7658,8 @@ Library - The functools module now has a total_ordering() class decorator to simplify the specification of rich comparisons. -- The functools module also adds cmp_to_key() as a tool to transition old- - style comparison functions to new-style key-functions. +- The functools module also adds cmp_to_key() as a tool to transition + old-style comparison functions to new-style key-functions. - bpo-8294: The Fraction constructor now accepts Decimal and float instances directly. @@ -7908,7 +8309,7 @@ Library - bpo-7470: logging: Fix bug in Unicode encoding fallback. - bpo-5949: Fixed IMAP4_SSL hang when the IMAP server response is missing - proper end-of- line termination. + proper end-of-line termination. - bpo-7457: Added a read_pkg_file method to ``distutils.dist.DistributionMetadata``. @@ -7937,8 +8338,8 @@ Build - bpo-6491: Allow --with-dbmliborder to specify that no dbms will be built. -- bpo-6943: Use pkg-config to find the libffi headers when the ``--with- - system-ffi`` flag is used. +- bpo-6943: Use pkg-config to find the libffi headers when the + ``--with-system-ffi`` flag is used. - bpo-7609: Add a ``--with-system-expat`` option that causes the system's expat library to be used for the pyexpat module instead of the one @@ -8006,7 +8407,7 @@ Core and Builtins and complex constructors; old-style and new-style numeric formatting; serialization and deserialization of floats and complex numbers using marshal, pickle and json; parsing of float and imaginary literals in - Python code; Decimal-to- float conversion. + Python code; Decimal-to-float conversion. The conversions use a Python-adapted version of David Gay's well-known dtoa.c, providing correctly-rounded strtod and dtoa C functions. This @@ -8319,8 +8720,8 @@ Core and Builtins - bpo-4074: Change the criteria for doing a full garbage collection (i.e. collecting the oldest generation) so that allocating lots of objects without destroying them does not show quadratic performance. Based on a - proposal by Martin von Löwis at http://mail.python.org/pipermail/python- - dev/2008-June/080579.html. + proposal by Martin von Löwis at + http://mail.python.org/pipermail/python-dev/2008-June/080579.html. - bpo-4850: Change COUNT_ALLOCS variables to Py_ssize_t. @@ -9207,8 +9608,8 @@ Library - bpo-1276768: The verbose option was not used in the code of distutils.file_util and distutils.dir_util. -- bpo-5132: Fixed trouble building extensions under Solaris with --enabled- - shared activated. Initial patch by Dave Peterson. +- bpo-5132: Fixed trouble building extensions under Solaris with + --enabled-shared activated. Initial patch by Dave Peterson. - bpo-1581476: Always use the Tcl global namespace when calling into Tcl. @@ -9235,8 +9636,8 @@ Library - Added new itertools functions: combinations_with_replacement() and compress(). -- bpo-5032: Added a step argument to itertools.count() and allowed non- - integer arguments. +- bpo-5032: Added a step argument to itertools.count() and allowed + non-integer arguments. - Fix and properly document the multiprocessing module's logging support, expose the internal levels and provide proper usage examples. @@ -9250,9 +9651,9 @@ Library - bpo-4710: Extract directories properly in the zipfile module; allow adding directories to a zipfile. -- bpo-3807: _multiprocessing build fails when configure is passed --without- - threads argument. When this occurs, _multiprocessing will be disabled, and - not compiled. +- bpo-3807: _multiprocessing build fails when configure is passed + --without-threads argument. When this occurs, _multiprocessing will be + disabled, and not compiled. - bpo-5008: When a file is opened in append mode with the new IO library, do an explicit seek to the end of file (so that e.g. tell() returns the file @@ -9373,8 +9774,8 @@ Library correct results in the case where one argument is a quiet NaN and the other is a finite number that requires rounding. -- bpo-1030250: Distutils created directories even when run with the --dry- - run option. +- bpo-1030250: Distutils created directories even when run with the + --dry-run option. - bpo-4483: _dbm module now builds on systems with gdbm & gdbm_compat libs. @@ -9719,10 +10120,10 @@ Tests - bpo-7270: Add some dedicated unit tests for multi-thread synchronization primitives such as Lock, RLock, Condition, Event and Semaphore. -- bpo-7222: Make thread "reaping" more reliable so that reference leak- - chasing test runs give sensible results. The previous method of reaping - threads could return successfully while some Thread objects were still - referenced. This also introduces a new private function: +- bpo-7222: Make thread "reaping" more reliable so that reference + leak-chasing test runs give sensible results. The previous method of + reaping threads could return successfully while some Thread objects were + still referenced. This also introduces a new private function: ``thread._count()``. - bpo-7151: Fixed regrtest -j so that output to stderr from a test no longer @@ -9741,7 +10142,7 @@ Tests off the trailing 0 in the version number. - bpo-5450: Moved tests involving loading tk from Lib/test/test_tcl to - Lib/lib- tk/test/test_tkinter/test_loadtk. With this, these tests + Lib/lib-tk/test/test_tkinter/test_loadtk. With this, these tests demonstrate the same behaviour as test_ttkguionly (and now also test_tk) which is to skip the tests if DISPLAY is defined but can't be used. @@ -10764,8 +11165,8 @@ Library code has been kept as backup). This will allow for using the 'warning's machinery in such places as the parser where use of pure Python code is not possible. Both the ``showarning()`` and ``formatwarning()`` gain an - optional 'line' argument which is not called by default for backwards- - compatibility reasons. Setting ``warnings.showwarning()`` to an + optional 'line' argument which is not called by default for + backwards-compatibility reasons. Setting ``warnings.showwarning()`` to an implementation that lacks support for the ``line`` argument will raise a DeprecationWarning. @@ -10918,7 +11319,7 @@ Core and Builtins \Uxxxxxxxx did not accept values outside the Basic Multilingual Plane. This affected raw unicode literals and the 'raw-unicode-escape' codec. Now UTF-16 surrogates are generated in this case, like normal unicode - literals and the 'unicode- escape' codec. + literals and the 'unicode-escape' codec. - bpo-2348: add Py3k warning for file.softspace. @@ -10953,8 +11354,9 @@ Core and Builtins - bpo-2359: Adding deprecation warnings for array.{read,write}. -- bpo-1779871: GNU gcc can now build Python on OS X because the flags -Wno- - long-double, -no-cpp-precomp, and -mno-fused-madd are no longer passed. +- bpo-1779871: GNU gcc can now build Python on OS X because the flags + -Wno-long-double, -no-cpp-precomp, and -mno-fused-madd are no longer + passed. - Add a warning when asserting a non-empty tuple which is always true. @@ -11002,7 +11404,7 @@ Library - bpo-1681432: Add triangular distribution to the random module - bpo-2136: urllib2's auth handler now allows single-quoted realms in the - WWW- Authenticate header. + WWW-Authenticate header. - bpo-2434: Enhanced platform.win32_ver() to also work on Python installation which do not have the win32all package installed. @@ -11300,7 +11702,7 @@ Core and Builtins automatically shifts from ints to longs. - bpo-1686386: Tuple's tp_repr did not take into account the possibility of - having a self- referential tuple, which is possible from C code. Nor did + having a self-referential tuple, which is possible from C code. Nor did object's tp_str consider that a type's tp_str could do something that could lead to an inifinite recursion. Py_ReprEnter() and Py_EnterRecursiveCall(), respectively, fixed the issues. @@ -11395,8 +11797,8 @@ Core and Builtins argument, if it exists. If not, it will work like before. This allows customizing the output of dir() in the presence of a __getattr__(). -- bpo-922167: Python no longer segfaults when faced with infinitely self- - recursive reload() calls (as reported by bug #742342). +- bpo-922167: Python no longer segfaults when faced with infinitely + self-recursive reload() calls (as reported by bug #742342). - bpo-1675981: remove unreachable code from ``type.__new__()`` method. @@ -11610,7 +12012,7 @@ Library - bpo-1530959: distutils' build command now uses different build directory when building extension modules against versions of Python compiled with - ``--with- pydebug``. + ``--with-pydebug``. - bpo-1555501: move plistlib from plat-mac directory to general library. @@ -11645,8 +12047,8 @@ Library - bpo-1693149: trace.py --ignore-module - accept multiple comma-separated modules to be given. -- bpo-1822: MIMEMultipart.is_multipart() behaves correctly for a just- - created (and empty) instance. Thanks Jonathan Share. +- bpo-1822: MIMEMultipart.is_multipart() behaves correctly for a + just-created (and empty) instance. Thanks Jonathan Share. - bpo-1861: Added an attribute to the sched module which returns an ordered list of upcoming events (displayed as named tuples). @@ -11756,7 +12158,7 @@ Library with --with-system-ffi. - bpo-1203: ctypes now does work on OS X when Python is built with - --disable-toolbox- glue. + ``--disable-toolbox-glue``. - collections.deque() now supports a "maxlen" argument. @@ -12665,7 +13067,7 @@ Build platform. - bpo-1655392: don't add -L/usr/lib/pythonX.Y/config to the LDFLAGS returned - by python- config if Python was built with --enable-shared because that + by python-config if Python was built with --enable-shared because that prevented the shared library from being used. - bpo-1569798: fix a bug in distutils when building Python from a directory diff --git a/Modules/Setup.dist b/Modules/Setup.dist index 523e05a..bbc9222 100644 --- a/Modules/Setup.dist +++ b/Modules/Setup.dist @@ -467,17 +467,9 @@ GLHACK=-Dclear=__GLclear #zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz # Interface to the Expat XML parser -# -# Expat was written by James Clark and is now maintained by a group of -# developers on SourceForge; see www.libexpat.org for more -# information. The pyexpat module was written by Paul Prescod after a -# prototype by Jack Jansen. Source of Expat 1.95.2 is included in -# Modules/expat/. Usage of a system shared libexpat.so/expat.dll is -# not advised. -# # More information on Expat can be found at www.libexpat.org. # -#pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI +#pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI # Hye-Shik Chang's CJKCodecs diff --git a/Modules/_bsddb.c b/Modules/_bsddb.c index 9c81ec5..cee447b 100644 --- a/Modules/_bsddb.c +++ b/Modules/_bsddb.c @@ -1503,56 +1503,71 @@ _db_associateCallback(DB* db, const DBT* priKey, const DBT* priData, else if (PyList_Check(result)) { char* data; - Py_ssize_t size; - int i, listlen; + Py_ssize_t size, listlen, i; DBT* dbts; listlen = PyList_Size(result); - dbts = (DBT *)malloc(sizeof(DBT) * listlen); - - for (i=0; i<listlen; i++) - { - if (!PyBytes_Check(PyList_GetItem(result, i))) - { - PyErr_SetString( - PyExc_TypeError, + if (listlen > PY_SIZE_MAX / sizeof(DBT)) { + PyErr_NoMemory(); + PyErr_Print(); + } + else { + dbts = (DBT *)malloc(sizeof(DBT) * listlen); + if (dbts == NULL) { + PyErr_NoMemory(); + PyErr_Print(); + } + else { + for (i = 0; i < listlen; i++) { + if (!PyBytes_Check(PyList_GetItem(result, i))) { + PyErr_SetString(PyExc_TypeError, #if (PY_VERSION_HEX < 0x03000000) "The list returned by DB->associate callback should be a list of strings."); #else "The list returned by DB->associate callback should be a list of bytes."); #endif - PyErr_Print(); - } - - PyBytes_AsStringAndSize( - PyList_GetItem(result, i), - &data, &size); - - CLEAR_DBT(dbts[i]); - dbts[i].data = malloc(size); /* TODO, check this */ - - if (dbts[i].data) - { - memcpy(dbts[i].data, data, size); - dbts[i].size = size; - dbts[i].ulen = dbts[i].size; - dbts[i].flags = DB_DBT_APPMALLOC; /* DB will free */ - } - else - { - PyErr_SetString(PyExc_MemoryError, - "malloc failed in _db_associateCallback (list)"); - PyErr_Print(); + break; + } + + if (PyBytes_AsStringAndSize(PyList_GetItem(result, i), + &data, &size) < 0) { + break; + } + + CLEAR_DBT(dbts[i]); + dbts[i].data = malloc(size); + if (dbts[i].data) { + memcpy(dbts[i].data, data, size); + dbts[i].size = size; + dbts[i].ulen = dbts[i].size; + /* DB will free. */ + dbts[i].flags = DB_DBT_APPMALLOC; + } + else { + PyErr_SetString(PyExc_MemoryError, + "malloc failed in " + "_db_associateCallback (list)"); + break; + } + } + if (PyErr_Occurred()) { + PyErr_Print(); + while (i--) { + free(dbts[i].data); + } + free(dbts); + } + else { + CLEAR_DBT(*secKey); + + secKey->data = dbts; + secKey->size = listlen; + secKey->flags = DB_DBT_APPMALLOC | DB_DBT_MULTIPLE; + retval = 0; + } } } - - CLEAR_DBT(*secKey); - - secKey->data = dbts; - secKey->size = listlen; - secKey->flags = DB_DBT_APPMALLOC | DB_DBT_MULTIPLE; - retval = 0; } #endif else { @@ -2242,7 +2257,7 @@ static PyObject* DB_join(DBObject* self, PyObject* args) { int err, flags=0; - int length, x; + Py_ssize_t length, x; PyObject* cursorsObj; DBC** cursors; DBC* dbc; @@ -2259,6 +2274,12 @@ DB_join(DBObject* self, PyObject* args) } length = PyObject_Length(cursorsObj); + if (length == -1) { + return NULL; + } + if (length >= PY_SSIZE_T_MAX / sizeof(DBC*)) { + return PyErr_NoMemory(); + } cursors = malloc((length+1) * sizeof(DBC*)); if (!cursors) { PyErr_NoMemory(); @@ -2276,6 +2297,7 @@ DB_join(DBObject* self, PyObject* args) PyErr_SetString(PyExc_TypeError, "Sequence of DBCursor objects expected"); free(cursors); + Py_DECREF(item); return NULL; } cursors[x] = ((DBCursorObject*)item)->dbc; diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 2e4da4c..3ca3930 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -639,7 +639,7 @@ deque_remove(dequeobject *deque, PyObject *value) PyDoc_STRVAR(remove_doc, "D.remove(value) -- remove first occurrence of value."); -static void +static int deque_clear(dequeobject *deque) { block *b; @@ -650,7 +650,7 @@ deque_clear(dequeobject *deque) PyObject *item; if (deque->len == 0) - return; + return 0; /* During the process of clearing a deque, decrefs can cause the deque to mutate. To avoid fatal confusion, we have to make the @@ -701,7 +701,7 @@ deque_clear(dequeobject *deque) } assert(leftblock->rightlink == NULL); freeblock(leftblock); - return; + return 0; alternate_method: while (deque->len) { @@ -709,6 +709,7 @@ deque_clear(dequeobject *deque) assert (item != NULL); Py_DECREF(item); } + return 0; } static PyObject * @@ -858,11 +859,20 @@ deque_traverse(dequeobject *deque, visitproc visit, void *arg) static PyObject * deque_copy(PyObject *deque) { + PyObject *result; if (((dequeobject *)deque)->maxlen == -1) - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "O", deque, NULL); + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "O", deque, NULL); else - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", deque, ((dequeobject *)deque)->maxlen, NULL); + if (result != NULL && !PyObject_TypeCheck(result, &deque_type)) { + PyErr_Format(PyExc_TypeError, + "%.200s() must return a deque, not %.200s", + Py_TYPE(deque)->tp_name, Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; } PyDoc_STRVAR(copy_doc, "Return a shallow copy of a deque."); diff --git a/Modules/_csv.c b/Modules/_csv.c index c39c0f1..88e3e90 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -224,7 +224,7 @@ _set_int(const char *name, int *target, PyObject *src, int dflt) if (src == NULL) *target = dflt; else { - if (!PyInt_Check(src) && !PyLong_Check(src)) { + if (!_PyAnyInt_Check(src)) { PyErr_Format(PyExc_TypeError, "\"%s\" must be an integer", name); return -1; @@ -565,24 +565,23 @@ parse_save_field(ReaderObj *self) static int parse_grow_buff(ReaderObj *self) { - if (self->field_size == 0) { - self->field_size = 4096; - if (self->field != NULL) - PyMem_Free(self->field); - self->field = PyMem_Malloc(self->field_size); - } - else { - if (self->field_size > INT_MAX / 2) { - PyErr_NoMemory(); - return 0; - } - self->field_size *= 2; - self->field = PyMem_Realloc(self->field, self->field_size); + unsigned field_size_new; + char *field_new; + + assert((unsigned)self->field_size <= INT_MAX); + + field_size_new = self->field_size ? 2 * (unsigned)self->field_size : 4096; + if (field_size_new > INT_MAX) { + PyErr_NoMemory(); + return 0; } - if (self->field == NULL) { + field_new = (char *)PyMem_Realloc(self->field, field_size_new); + if (field_new == NULL) { PyErr_NoMemory(); return 0; } + self->field = field_new; + self->field_size = (int)field_size_new; return 1; } @@ -1088,31 +1087,24 @@ join_append_data(WriterObj *self, char *field, int quote_empty, static int join_check_rec_size(WriterObj *self, int rec_len) { + unsigned rec_size_new; + char *rec_new; - if (rec_len < 0 || rec_len > INT_MAX - MEM_INCR) { - PyErr_NoMemory(); - return 0; - } + assert(rec_len >= 0); if (rec_len > self->rec_size) { - if (self->rec_size == 0) { - self->rec_size = (rec_len / MEM_INCR + 1) * MEM_INCR; - if (self->rec != NULL) - PyMem_Free(self->rec); - self->rec = PyMem_Malloc(self->rec_size); - } - else { - char *old_rec = self->rec; - - self->rec_size = (rec_len / MEM_INCR + 1) * MEM_INCR; - self->rec = PyMem_Realloc(self->rec, self->rec_size); - if (self->rec == NULL) - PyMem_Free(old_rec); + rec_size_new = (unsigned)(rec_len / MEM_INCR + 1) * MEM_INCR; + if (rec_size_new > INT_MAX) { + PyErr_NoMemory(); + return 0; } - if (self->rec == NULL) { + rec_new = (char *)PyMem_Realloc(self->rec, rec_size_new); + if (rec_new == NULL) { PyErr_NoMemory(); return 0; } + self->rec = rec_new; + self->rec_size = (int)rec_size_new; } return 1; } @@ -1452,7 +1444,7 @@ csv_field_size_limit(PyObject *module, PyObject *args) if (!PyArg_UnpackTuple(args, "field_size_limit", 0, 1, &new_limit)) return NULL; if (new_limit != NULL) { - if (!PyInt_Check(new_limit) && !PyLong_Check(new_limit)) { + if (!_PyAnyInt_Check(new_limit)) { PyErr_Format(PyExc_TypeError, "limit must be an integer"); return NULL; diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 25740ed..8abcd30 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -547,7 +547,7 @@ static PyObject * CDataType_from_address(PyObject *type, PyObject *value) { void *buf; - if (!PyInt_Check(value) && !PyLong_Check(value)) { + if (!_PyAnyInt_Check(value)) { PyErr_SetString(PyExc_TypeError, "integer expected"); return NULL; @@ -691,7 +691,7 @@ CDataType_in_dll(PyObject *type, PyObject *args) obj = PyObject_GetAttrString(dll, "_handle"); if (!obj) return NULL; - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!_PyAnyInt_Check(obj)) { PyErr_SetString(PyExc_TypeError, "the _handle attribute of the second argument must be an integer"); Py_DECREF(obj); @@ -1216,6 +1216,10 @@ CharArray_set_raw(CDataObject *self, PyObject *value) #if (PY_VERSION_HEX >= 0x02060000) Py_buffer view = { 0 }; #endif + if (value == NULL) { + PyErr_SetString(PyExc_AttributeError, "cannot delete attribute"); + return -1; + } if (PyBuffer_Check(value)) { size = Py_TYPE(value)->tp_as_buffer->bf_getreadbuffer(value, 0, (void *)&ptr); if (size < 0) @@ -1460,24 +1464,36 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) PyTypeObject *result; StgDictObject *stgdict; StgDictObject *itemdict; - PyObject *proto; + PyObject *proto, *length_attr; PyObject *typedict; - long length; - + Py_ssize_t length; Py_ssize_t itemsize, itemalign; typedict = PyTuple_GetItem(args, 2); if (!typedict) return NULL; - proto = PyDict_GetItemString(typedict, "_length_"); /* Borrowed ref */ - if (!proto || !PyInt_Check(proto)) { + length_attr = PyDict_GetItemString(typedict, "_length_"); /* Borrowed ref */ + if (!length_attr || !_PyAnyInt_Check(length_attr)) { PyErr_SetString(PyExc_AttributeError, "class must define a '_length_' attribute, " "which must be a positive integer"); return NULL; } - length = PyInt_AS_LONG(proto); + if (PyInt_Check(length_attr)) { + length = PyInt_AS_LONG(length_attr); + } + else { + assert(PyLong_Check(length_attr)); + length = PyLong_AsSsize_t(length_attr); + if (length == -1 && PyErr_Occurred()) { + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + PyErr_SetString(PyExc_OverflowError, + "The '_length_' attribute is too large"); + } + return NULL; + } + } proto = PyDict_GetItemString(typedict, "_type_"); /* Borrowed ref */ if (!proto) { @@ -1779,7 +1795,7 @@ c_void_p_from_param(PyObject *type, PyObject *value) } /* Should probably allow buffer interface as well */ /* int, long */ - if (PyInt_Check(value) || PyLong_Check(value)) { + if (_PyAnyInt_Check(value)) { PyCArgObject *parg; struct fielddesc *fd = _ctypes_get_fielddesc("P"); @@ -3419,7 +3435,7 @@ static int _get_name(PyObject *obj, char **pname) { #ifdef MS_WIN32 - if (PyInt_Check(obj) || PyLong_Check(obj)) { + if (_PyAnyInt_Check(obj)) { /* We have to use MAKEINTRESOURCEA for Windows CE. Works on Windows as well, of course. */ @@ -3469,7 +3485,7 @@ PyCFuncPtr_FromDll(PyTypeObject *type, PyObject *args, PyObject *kwds) Py_DECREF(ftuple); return NULL; } - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!_PyAnyInt_Check(obj)) { PyErr_SetString(PyExc_TypeError, "the _handle attribute of the second argument must be an integer"); Py_DECREF(ftuple); @@ -3514,20 +3530,23 @@ PyCFuncPtr_FromDll(PyTypeObject *type, PyObject *args, PyObject *kwds) return NULL; } #endif - Py_INCREF(dll); /* for KeepRef */ - Py_DECREF(ftuple); - if (!_validate_paramflags(type, paramflags)) + if (!_validate_paramflags(type, paramflags)) { + Py_DECREF(ftuple); return NULL; + } self = (PyCFuncPtrObject *)GenericPyCData_new(type, args, kwds); - if (!self) + if (!self) { + Py_DECREF(ftuple); return NULL; + } Py_XINCREF(paramflags); self->paramflags = paramflags; *(void **)self->b_ptr = address; - + Py_INCREF(dll); + Py_DECREF(ftuple); if (-1 == KeepRef((CDataObject *)self, 0, dll)) { Py_DECREF((PyObject *)self); return NULL; @@ -3600,8 +3619,7 @@ PyCFuncPtr_new(PyTypeObject *type, PyObject *args, PyObject *kwds) #endif if (1 == PyTuple_GET_SIZE(args) - && (PyInt_Check(PyTuple_GET_ITEM(args, 0)) - || PyLong_Check(PyTuple_GET_ITEM(args, 0)))) { + && _PyAnyInt_Check(PyTuple_GET_ITEM(args, 0))) { CDataObject *ob; void *ptr = PyLong_AsVoidPtr(PyTuple_GET_ITEM(args, 0)); if (ptr == NULL && PyErr_Occurred()) diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c index f295c6f..93876d3 100644 --- a/Modules/_ctypes/_ctypes_test.c +++ b/Modules/_ctypes/_ctypes_test.c @@ -52,6 +52,19 @@ _testfunc_cbk_large_struct(Test in, void (*func)(Test)) func(in); } +/* + * See issue 29565. Update a structure passed by value; + * the caller should not see any change. + */ + +EXPORT(void) +_testfunc_large_struct_update_value(Test in) +{ + in.first = 0x0badf00d; + in.second = 0x0badf00d; + in.third = 0x0badf00d; +} + EXPORT(void)testfunc_array(int values[4]) { printf("testfunc_array %d %d %d %d\n", @@ -643,6 +656,200 @@ EXPORT(void) TwoOutArgs(int a, int *pi, int b, int *pj) } #ifdef MS_WIN32 + +typedef struct { + char f1; +} Size1; + +typedef struct { + char f1; + char f2; +} Size2; + +typedef struct { + char f1; + char f2; + char f3; +} Size3; + +typedef struct { + char f1; + char f2; + char f3; + char f4; +} Size4; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; +} Size5; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; + char f6; +} Size6; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; + char f6; + char f7; +} Size7; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; + char f6; + char f7; + char f8; +} Size8; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; + char f6; + char f7; + char f8; + char f9; +} Size9; + +typedef struct { + char f1; + char f2; + char f3; + char f4; + char f5; + char f6; + char f7; + char f8; + char f9; + char f10; +} Size10; + +EXPORT(Size1) TestSize1() { + Size1 f; + f.f1 = 'a'; + return f; +} + +EXPORT(Size2) TestSize2() { + Size2 f; + f.f1 = 'a'; + f.f2 = 'b'; + return f; +} + +EXPORT(Size3) TestSize3() { + Size3 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + return f; +} + +EXPORT(Size4) TestSize4() { + Size4 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + return f; +} + +EXPORT(Size5) TestSize5() { + Size5 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + return f; +} + +EXPORT(Size6) TestSize6() { + Size6 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + f.f6 = 'f'; + return f; +} + +EXPORT(Size7) TestSize7() { + Size7 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + f.f6 = 'f'; + f.f7 = 'g'; + return f; +} + +EXPORT(Size8) TestSize8() { + Size8 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + f.f6 = 'f'; + f.f7 = 'g'; + f.f8 = 'h'; + return f; +} + +EXPORT(Size9) TestSize9() { + Size9 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + f.f6 = 'f'; + f.f7 = 'g'; + f.f8 = 'h'; + f.f9 = 'i'; + return f; +} + +EXPORT(Size10) TestSize10() { + Size10 f; + f.f1 = 'a'; + f.f2 = 'b'; + f.f3 = 'c'; + f.f4 = 'd'; + f.f5 = 'e'; + f.f6 = 'f'; + f.f7 = 'g'; + f.f8 = 'h'; + f.f9 = 'i'; + f.f10 = 'j'; + return f; +} + +#endif + +#ifdef MS_WIN32 EXPORT(S2H) __stdcall s_ret_2h_func(S2H inp) { return ret_2h_func(inp); } EXPORT(S8I) __stdcall s_ret_8i_func(S8I inp) { return ret_8i_func(inp); } #endif diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 91233d5..2097342 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -485,11 +485,7 @@ PyCArg_repr(PyCArgObject *self) case 'q': case 'Q': sprintf(buffer, -#ifdef MS_WIN32 - "<cparam '%c' (%I64d)>", -#else - "<cparam '%c' (%qd)>", -#endif + "<cparam '%c' (%" PY_FORMAT_LONG_LONG "d)>", self->tag, self->value.q); break; #endif @@ -747,9 +743,9 @@ ffi_type *_ctypes_get_ffi_type(PyObject *obj) It returns small structures in registers */ if (dict->ffi_type_pointer.type == FFI_TYPE_STRUCT) { - if (dict->ffi_type_pointer.size <= 4) + if (can_return_struct_as_int(dict->ffi_type_pointer.size)) return &ffi_type_sint32; - else if (dict->ffi_type_pointer.size <= 8) + else if (can_return_struct_as_sint64 (dict->ffi_type_pointer.size)) return &ffi_type_sint64; } #endif diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 6f632d0..46f041b 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1361,7 +1361,7 @@ z_set(void *ptr, PyObject *value, Py_ssize_t size) return NULL; *(char **)ptr = PyString_AS_STRING(str); return str; - } else if (PyInt_Check(value) || PyLong_Check(value)) { + } else if (_PyAnyInt_Check(value)) { #if SIZEOF_VOID_P == SIZEOF_LONG_LONG *(char **)ptr = (char *)PyInt_AsUnsignedLongLongMask(value); #else @@ -1410,7 +1410,7 @@ Z_set(void *ptr, PyObject *value, Py_ssize_t size) _ctypes_conversion_errors); if (!value) return NULL; - } else if (PyInt_Check(value) || PyLong_Check(value)) { + } else if (_PyAnyInt_Check(value)) { #if SIZEOF_VOID_P == SIZEOF_LONG_LONG *(wchar_t **)ptr = (wchar_t *)PyInt_AsUnsignedLongLongMask(value); #else @@ -1565,7 +1565,7 @@ P_set(void *ptr, PyObject *value, Py_ssize_t size) _RET(value); } - if (!PyInt_Check(value) && !PyLong_Check(value)) { + if (!_PyAnyInt_Check(value)) { PyErr_SetString(PyExc_TypeError, "cannot be converted to pointer"); return NULL; diff --git a/Modules/_ctypes/libffi_msvc/ffi.c b/Modules/_ctypes/libffi_msvc/ffi.c index 515d802..15a100f 100644 --- a/Modules/_ctypes/libffi_msvc/ffi.c +++ b/Modules/_ctypes/libffi_msvc/ffi.c @@ -119,13 +119,28 @@ void ffi_prep_args(char *stack, extended_cif *ecif) argp += z; } - if (argp - stack > ecif->cif->bytes) + if (argp >= stack && (unsigned)(argp - stack) > ecif->cif->bytes) { Py_FatalError("FFI BUG: not enough stack space for arguments"); } return; } +/* +Per: https://msdn.microsoft.com/en-us/library/7572ztz4.aspx +To be returned by value in RAX, user-defined types must have a length +of 1, 2, 4, 8, 16, 32, or 64 bits +*/ +int can_return_struct_as_int(size_t s) +{ + return s == 1 || s == 2 || s == 4; +} + +int can_return_struct_as_sint64(size_t s) +{ + return s == 8; +} + /* Perform machine dependent cif processing */ ffi_status ffi_prep_cif_machdep(ffi_cif *cif) { @@ -144,9 +159,9 @@ ffi_status ffi_prep_cif_machdep(ffi_cif *cif) /* MSVC returns small structures in registers. Put in cif->flags the value FFI_TYPE_STRUCT only if the structure is big enough; otherwise, put the 4- or 8-bytes integer type. */ - if (cif->rtype->size <= 4) + if (can_return_struct_as_int(cif->rtype->size)) cif->flags = FFI_TYPE_INT; - else if (cif->rtype->size <= 8) + else if (can_return_struct_as_sint64(cif->rtype->size)) cif->flags = FFI_TYPE_SINT64; else cif->flags = FFI_TYPE_STRUCT; @@ -220,6 +235,20 @@ ffi_call(/*@dependent@*/ ffi_cif *cif, break; #else case FFI_SYSV: + /* use a local scope for the 'i' variable */ + { + unsigned i; + /* If a single argument takes more than 8 bytes, + then a copy is passed by reference. */ + for (i = 0; i < cif->nargs; i++) { + size_t z = cif->arg_types[i]->size; + if (z > 8) { + void *temp = alloca(z); + memcpy(temp, avalue[i], z); + avalue[i] = temp; + } + } + } /*@-usedef@*/ return ffi_call_AMD64(ffi_prep_args, &ecif, cif->bytes, cif->flags, ecif.rvalue, fn); diff --git a/Modules/_ctypes/libffi_msvc/ffi.h b/Modules/_ctypes/libffi_msvc/ffi.h index efb14c5..ba74202 100644 --- a/Modules/_ctypes/libffi_msvc/ffi.h +++ b/Modules/_ctypes/libffi_msvc/ffi.h @@ -136,6 +136,9 @@ typedef struct _ffi_type /*@null@*/ struct _ffi_type **elements; } ffi_type; +int can_return_struct_as_int(size_t); +int can_return_struct_as_sint64(size_t); + /* These are defined in types.c */ extern ffi_type ffi_type_void; extern ffi_type ffi_type_uint8; diff --git a/Modules/_ctypes/libffi_msvc/prep_cif.c b/Modules/_ctypes/libffi_msvc/prep_cif.c index b28f895..80c5a60 100644 --- a/Modules/_ctypes/libffi_msvc/prep_cif.c +++ b/Modules/_ctypes/libffi_msvc/prep_cif.c @@ -117,7 +117,8 @@ ffi_status ffi_prep_cif(/*@out@*/ /*@partial@*/ ffi_cif *cif, /* Make space for the return structure pointer */ if (cif->rtype->type == FFI_TYPE_STRUCT #ifdef _WIN32 - && (cif->rtype->size > 8) /* MSVC returns small structs in registers */ + && !can_return_struct_as_int(cif->rtype->size) /* MSVC returns small structs in registers */ + && !can_return_struct_as_sint64(cif->rtype->size) #endif #ifdef SPARC && (cif->abi != FFI_V9 || cif->rtype->size > 32) @@ -146,7 +147,9 @@ ffi_status ffi_prep_cif(/*@out@*/ /*@partial@*/ ffi_cif *cif, bytes += sizeof(void*); else #elif defined (_WIN64) - if ((*ptr)->type == FFI_TYPE_STRUCT && ((*ptr)->size > 8)) + if ((*ptr)->type == FFI_TYPE_STRUCT && + !can_return_struct_as_int((*ptr)->size) && + !can_return_struct_as_sint64((*ptr)->size)) bytes += sizeof(void*); else #endif diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 935712a..0ec4ee1 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -194,7 +194,7 @@ PyCursesCheckERR(int code, char *fname) static int PyCurses_ConvertToChtype(PyObject *obj, chtype *ch) { - if (PyInt_Check(obj) || PyLong_Check(obj)) { + if (_PyAnyInt_Check(obj)) { *ch = (chtype) PyInt_AsLong(obj); if (*ch == (chtype) -1 && PyErr_Occurred()) return 0; @@ -2603,7 +2603,7 @@ PyCurses_UnCtrl(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL; - if (PyInt_Check(temp) || PyLong_Check(temp)) { + if (_PyAnyInt_Check(temp)) { ch = (chtype) PyInt_AsLong(temp); if (ch == (chtype) -1 && PyErr_Occurred()) return NULL; @@ -2628,7 +2628,7 @@ PyCurses_UngetCh(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL; - if (PyInt_Check(temp) || PyLong_Check(temp)) { + if (_PyAnyInt_Check(temp)) { ch = (int) PyInt_AsLong(temp); if (ch == -1 && PyErr_Occurred()) return NULL; diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c index 1d316a1..bbaf994 100644 --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -499,8 +499,10 @@ element(PyObject* self, PyObject* args, PyObject* kw) attrib = (attrib) ? PyDict_Copy(attrib) : PyDict_New(); if (!attrib) return NULL; - if (kw) - PyDict_Update(attrib, kw); + if (kw != NULL && PyDict_Update(attrib, kw) < 0) { + Py_DECREF(attrib); + return NULL; + } } else { Py_INCREF(Py_None); attrib = Py_None; @@ -530,8 +532,10 @@ subelement(PyObject* self, PyObject* args, PyObject* kw) attrib = (attrib) ? PyDict_Copy(attrib) : PyDict_New(); if (!attrib) return NULL; - if (kw) - PyDict_Update(attrib, kw); + if (kw != NULL && PyDict_Update(attrib, kw) < 0) { + Py_DECREF(attrib); + return NULL; + } } else { Py_INCREF(Py_None); attrib = Py_None; @@ -1347,7 +1351,7 @@ element_subscr(PyObject* self_, PyObject* item) ElementObject* self = (ElementObject*) self_; #if (PY_VERSION_HEX < 0x02050000) - if (PyInt_Check(item) || PyLong_Check(item)) { + if (_PyAnyInt_Check(item)) { long i = PyInt_AsLong(item); #else if (PyIndex_Check(item)) { @@ -1404,7 +1408,7 @@ element_ass_subscr(PyObject* self_, PyObject* item, PyObject* value) ElementObject* self = (ElementObject*) self_; #if (PY_VERSION_HEX < 0x02050000) - if (PyInt_Check(item) || PyLong_Check(item)) { + if (_PyAnyInt_Check(item)) { long i = PyInt_AsLong(item); #else if (PyIndex_Check(item)) { @@ -1727,12 +1731,16 @@ treebuilder_new(void) self->data = NULL; - self->stack = PyList_New(20); self->index = 0; self->events = NULL; self->start_event_obj = self->end_event_obj = NULL; self->start_ns_event_obj = self->end_ns_event_obj = NULL; + self->stack = PyList_New(20); + if (self->stack == NULL) { + Py_DECREF(self); + return NULL; + } ALLOC(sizeof(TreeBuilderObject), "create treebuilder"); @@ -1756,7 +1764,7 @@ treebuilder_dealloc(TreeBuilderObject* self) Py_XDECREF(self->end_event_obj); Py_XDECREF(self->start_event_obj); Py_XDECREF(self->events); - Py_DECREF(self->stack); + Py_XDECREF(self->stack); Py_XDECREF(self->data); Py_DECREF(self->last); Py_DECREF(self->this); @@ -2549,16 +2557,28 @@ xmlparser(PyObject* self_, PyObject* args, PyObject* kw) if (self == NULL) return NULL; + /* Init to NULL to keep the error handling below manageable. */ + self->parser = NULL; + self->names = + self->target = + self->handle_xml = + self->handle_start = + self->handle_data = + self->handle_end = + self->handle_comment = + self->handle_pi = + self->handle_close = + NULL; + self->entity = PyDict_New(); if (!self->entity) { - PyObject_Del(self); + Py_DECREF(self); return NULL; } - + self->names = PyDict_New(); if (!self->names) { - PyObject_Del(self->entity); - PyObject_Del(self); + Py_DECREF(self); return NULL; } @@ -2568,26 +2588,18 @@ xmlparser(PyObject* self_, PyObject* args, PyObject* kw) self->parser = EXPAT(ParserCreate_MM)(encoding, &memory_handler, "}"); if (!self->parser) { - PyObject_Del(self->names); - PyObject_Del(self->entity); - PyObject_Del(self); + Py_DECREF(self); PyErr_NoMemory(); return NULL; } + /* expat < 2.1.0 has no XML_SetHashSalt() */ + if (EXPAT(SetHashSalt) != NULL) { + EXPAT(SetHashSalt)(self->parser, + (unsigned long)_Py_HashSecret.prefix); + } ALLOC(sizeof(XMLParserObject), "create expatparser"); - /* Init to NULL to keep the error handling below manageable. */ - self->target = - self->handle_xml = - self->handle_start = - self->handle_data = - self->handle_end = - self->handle_comment = - self->handle_pi = - self->handle_close = - NULL; - /* setup target handlers */ if (!target) { target = treebuilder_new(); @@ -2673,7 +2685,9 @@ xmlparser(PyObject* self_, PyObject* args, PyObject* kw) static void xmlparser_dealloc(XMLParserObject* self) { - EXPAT(ParserFree)(self->parser); + if (self->parser != NULL) { + EXPAT(ParserFree)(self->parser); + } Py_XDECREF(self->handle_close); Py_XDECREF(self->handle_pi); @@ -2683,9 +2697,9 @@ xmlparser_dealloc(XMLParserObject* self) Py_XDECREF(self->handle_start); Py_XDECREF(self->handle_xml); - Py_DECREF(self->target); - Py_DECREF(self->entity); - Py_DECREF(self->names); + Py_XDECREF(self->target); + Py_XDECREF(self->entity); + Py_XDECREF(self->names); RELEASE(sizeof(XMLParserObject), "destroy expatparser"); diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 5bef746..b8c98a4 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -1812,6 +1812,13 @@ bufferedwriter_write(buffered *self, PyObject *args) if (!PyArg_ParseTuple(args, "s*:write", &buf)) { return NULL; } + if (PyUnicode_Check(PyTuple_GET_ITEM(args, 0)) && + PyErr_WarnPy3k("write() argument must be string or buffer, " + "not 'unicode'", 1) < 0) + { + PyBuffer_Release(&buf); + return NULL; + } if (IS_CLOSED(self)) { PyErr_SetString(PyExc_ValueError, "write to closed file"); diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index 2b40ada..ed5b918 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -716,8 +716,16 @@ fileio_write(fileio *self, PyObject *args) if (!self->writable) return err_mode("writing"); - if (!PyArg_ParseTuple(args, "s*", &pbuf)) + if (!PyArg_ParseTuple(args, "s*:write", &pbuf)) { return NULL; + } + if (PyUnicode_Check(PyTuple_GET_ITEM(args, 0)) && + PyErr_WarnPy3k("write() argument must be string or buffer, " + "not 'unicode'", 1) < 0) + { + PyBuffer_Release(&pbuf); + return NULL; + } if (_PyVerify_fd(self->fd)) { Py_BEGIN_ALLOW_THREADS diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c index d813daf..17b905a 100644 --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -177,17 +177,25 @@ _PyIOBase_check_closed(PyObject *self, PyObject *args) static PyObject * iobase_close(PyObject *self, PyObject *args) { - PyObject *res; + PyObject *res, *exc, *val, *tb; + int rc; if (IS_CLOSED(self)) Py_RETURN_NONE; res = PyObject_CallMethodObjArgs(self, _PyIO_str_flush, NULL); - PyObject_SetAttrString(self, "__IOBase_closed", Py_True); + + PyErr_Fetch(&exc, &val, &tb); + rc = PyObject_SetAttrString(self, "__IOBase_closed", Py_True); + _PyErr_ReplaceException(exc, val, tb); + if (rc < 0) { + Py_CLEAR(res); + } + if (res == NULL) { return NULL; } - Py_XDECREF(res); + Py_DECREF(res); Py_RETURN_NONE; } diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c index 83bb7ac..c52ca27 100644 --- a/Modules/_io/stringio.c +++ b/Modules/_io/stringio.c @@ -693,8 +693,10 @@ stringio_getstate(stringio *self) } else { dict = PyDict_Copy(self->dict); - if (dict == NULL) + if (dict == NULL) { + Py_DECREF(initvalue); return NULL; + } } state = Py_BuildValue("(OOnN)", initvalue, diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index bf37f72..12a12f9 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -221,7 +221,7 @@ incrementalnewlinedecoder_init(nldecoder_object *self, self->errors = errors; } - self->translate = translate; + self->translate = translate ? 1 : 0; self->seennl = 0; self->pendingcr = 0; @@ -707,6 +707,8 @@ typedef struct PyObject *dict; } textio; +static void +textiowrapper_set_decoded_chars(textio *self, PyObject *chars); /* A couple of specialized cases in order to bypass the slow incremental encoding methods for the most popular encodings. */ @@ -1329,6 +1331,7 @@ textiowrapper_write(textio *self, PyObject *args) Py_DECREF(ret); } + textiowrapper_set_decoded_chars(self, NULL); Py_CLEAR(self->snapshot); if (self->decoder) { @@ -1463,6 +1466,7 @@ textiowrapper_read_chunk(textio *self) /* At the snapshot point, len(dec_buffer) bytes before the read, the * next input to be decoded is dec_buffer + input_chunk. */ + PyObject *snapshot; PyObject *next_input = PyNumber_Add(dec_buffer, input_chunk); if (next_input == NULL) goto fail; @@ -1474,8 +1478,13 @@ textiowrapper_read_chunk(textio *self) Py_DECREF(next_input); goto fail; } + snapshot = Py_BuildValue("NN", dec_flags, next_input); + if (snapshot == NULL) { + dec_flags = NULL; + goto fail; + } + Py_XSETREF(self->snapshot, snapshot); Py_DECREF(dec_buffer); - Py_XSETREF(self->snapshot, Py_BuildValue("NN", dec_flags, next_input)); } Py_DECREF(input_chunk); @@ -1534,6 +1543,7 @@ textiowrapper_read(textio *self, PyObject *args) if (final == NULL) goto fail; + textiowrapper_set_decoded_chars(self, NULL); Py_CLEAR(self->snapshot); return final; } @@ -2009,6 +2019,7 @@ textiowrapper_seek(textio *self, PyObject *args) int whence = 0; PyObject *res; int cmp; + PyObject *snapshot; CHECK_ATTACHED(self); @@ -2145,11 +2156,11 @@ textiowrapper_seek(textio *self, PyObject *args) goto fail; } - self->snapshot = Py_BuildValue("iN", cookie.dec_flags, input_chunk); - if (self->snapshot == NULL) { - Py_DECREF(input_chunk); + snapshot = Py_BuildValue("iN", cookie.dec_flags, input_chunk); + if (snapshot == NULL) { goto fail; } + Py_XSETREF(self->snapshot, snapshot); decoded = PyObject_CallMethod(self->decoder, "decode", "Oi", input_chunk, (int)cookie.need_eof); @@ -2167,9 +2178,10 @@ textiowrapper_seek(textio *self, PyObject *args) self->decoded_chars_used = cookie.chars_to_skip; } else { - self->snapshot = Py_BuildValue("is", cookie.dec_flags, ""); - if (self->snapshot == NULL) + snapshot = Py_BuildValue("is", cookie.dec_flags, ""); + if (snapshot == NULL) goto fail; + Py_XSETREF(self->snapshot, snapshot); } /* Finally, reset the encoder (merely useful for proper BOM handling) */ @@ -2581,6 +2593,10 @@ textiowrapper_chunk_size_set(textio *self, PyObject *arg, void *context) { Py_ssize_t n; CHECK_ATTACHED_INT(self); + if (arg == NULL) { + PyErr_SetString(PyExc_AttributeError, "cannot delete attribute"); + return -1; + } n = PyNumber_AsSsize_t(arg, PyExc_TypeError); if (n == -1 && PyErr_Occurred()) return -1; diff --git a/Modules/_json.c b/Modules/_json.c index 39ec467..3a88882 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -1893,7 +1893,7 @@ _encoded_const(PyObject *obj) if (s_null == NULL) { s_null = PyString_InternFromString("null"); } - Py_INCREF(s_null); + Py_XINCREF(s_null); return s_null; } else if (obj == Py_True) { @@ -1901,7 +1901,7 @@ _encoded_const(PyObject *obj) if (s_true == NULL) { s_true = PyString_InternFromString("true"); } - Py_INCREF(s_true); + Py_XINCREF(s_true); return s_true; } else if (obj == Py_False) { @@ -1909,7 +1909,7 @@ _encoded_const(PyObject *obj) if (s_false == NULL) { s_false = PyString_InternFromString("false"); } - Py_INCREF(s_false); + Py_XINCREF(s_false); return s_false; } else { @@ -1981,7 +1981,7 @@ encoder_listencode_obj(PyEncoderObject *s, PyObject *rval, PyObject *obj, Py_ssi return -1; return _steal_list_append(rval, encoded); } - else if (PyInt_Check(obj) || PyLong_Check(obj)) { + else if (_PyAnyInt_Check(obj)) { PyObject *encoded = PyObject_Str(obj); if (encoded == NULL) return -1; @@ -2131,7 +2131,7 @@ encoder_listencode_dict(PyEncoderObject *s, PyObject *rval, PyObject *dct, Py_ss if (kstr == NULL) goto bail; } - else if (PyInt_Check(key) || PyLong_Check(key)) { + else if (_PyAnyInt_Check(key)) { kstr = PyObject_Str(key); if (kstr == NULL) goto bail; diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c index 7e2f1a9..c55bd0d 100644 --- a/Modules/_localemodule.c +++ b/Modules/_localemodule.c @@ -73,20 +73,13 @@ copy_grouping(char* s) do { i++; val = PyInt_FromLong(s[i]); - if (!val) - break; - if (PyList_SetItem(result, i, val)) { - Py_DECREF(val); - val = NULL; - break; + if (val == NULL) { + Py_DECREF(result); + return NULL; } + PyList_SET_ITEM(result, i, val); } while (s[i] != '\0' && s[i] != CHAR_MAX); - if (!val) { - Py_DECREF(result); - return NULL; - } - return result; } diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 6090c7d..945354f 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -807,11 +807,11 @@ static PyMethodDef profiler_methods[] = { }; PyDoc_STRVAR(profiler_doc, "\ -Profiler(custom_timer=None, time_unit=None, subcalls=True, builtins=True)\n\ +Profiler(timer=None, timeunit=None, subcalls=True, builtins=True)\n\ \n\ Builds a profiler object using the specified timer function.\n\ The default timer is a fast built-in one based on real time.\n\ - For custom timer functions returning integers, time_unit can\n\ + For custom timer functions returning integers, timeunit can\n\ be a float specifying a scale (i.e. how long each integer unit\n\ is, in seconds).\n\ "); diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c index 2e49db6..2c0daef 100644 --- a/Modules/_randommodule.c +++ b/Modules/_randommodule.c @@ -415,7 +415,7 @@ random_jumpahead(RandomObject *self, PyObject *n) PyObject *remobj; unsigned long *mt, tmp, nonzero; - if (!PyInt_Check(n) && !PyLong_Check(n)) { + if (!_PyAnyInt_Check(n)) { PyErr_Format(PyExc_TypeError, "jumpahead requires an " "integer, not '%s'", Py_TYPE(n)->tp_name); diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 0603b43..18a6b42 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -675,7 +675,7 @@ void _pysqlite_func_callback(sqlite3_context* context, int argc, sqlite3_value** Py_DECREF(py_retval); } if (!ok) { - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -711,7 +711,7 @@ static void _pysqlite_step_callback(sqlite3_context *context, int argc, sqlite3_ if (PyErr_Occurred()) { *aggregate_instance = 0; - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -735,7 +735,7 @@ static void _pysqlite_step_callback(sqlite3_context *context, int argc, sqlite3_ Py_DECREF(args); if (!function_result) { - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -781,7 +781,7 @@ void _pysqlite_final_callback(sqlite3_context* context) Py_DECREF(function_result); } if (!ok) { - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -877,19 +877,17 @@ PyObject* pysqlite_connection_create_function(pysqlite_Connection* self, PyObjec return NULL; } + if (PyDict_SetItem(self->function_pinboard, func, Py_None) == -1) { + return NULL; + } rc = sqlite3_create_function(self->db, name, narg, SQLITE_UTF8, (void*)func, _pysqlite_func_callback, NULL, NULL); if (rc != SQLITE_OK) { /* Workaround for SQLite bug: no error code or string is available here */ PyErr_SetString(pysqlite_OperationalError, "Error creating function"); return NULL; - } else { - if (PyDict_SetItem(self->function_pinboard, func, Py_None) == -1) - return NULL; - - Py_INCREF(Py_None); - return Py_None; } + Py_RETURN_NONE; } PyObject* pysqlite_connection_create_aggregate(pysqlite_Connection* self, PyObject* args, PyObject* kwargs) @@ -910,18 +908,16 @@ PyObject* pysqlite_connection_create_aggregate(pysqlite_Connection* self, PyObje return NULL; } + if (PyDict_SetItem(self->function_pinboard, aggregate_class, Py_None) == -1) { + return NULL; + } rc = sqlite3_create_function(self->db, name, n_arg, SQLITE_UTF8, (void*)aggregate_class, 0, &_pysqlite_step_callback, &_pysqlite_final_callback); if (rc != SQLITE_OK) { /* Workaround for SQLite bug: no error code or string is available here */ PyErr_SetString(pysqlite_OperationalError, "Error creating aggregate"); return NULL; - } else { - if (PyDict_SetItem(self->function_pinboard, aggregate_class, Py_None) == -1) - return NULL; - - Py_INCREF(Py_None); - return Py_None; } + Py_RETURN_NONE; } static int _authorizer_callback(void* user_arg, int action, const char* arg1, const char* arg2 , const char* dbname, const char* access_attempt_source) @@ -936,7 +932,7 @@ static int _authorizer_callback(void* user_arg, int action, const char* arg1, co ret = PyObject_CallFunction((PyObject*)user_arg, "issss", action, arg1, arg2, dbname, access_attempt_source); if (!ret) { - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -972,7 +968,7 @@ static int _progress_handler(void* user_arg) ret = PyObject_CallFunction((PyObject*)user_arg, ""); if (!ret) { - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); @@ -1007,18 +1003,15 @@ static PyObject* pysqlite_connection_set_authorizer(pysqlite_Connection* self, P return NULL; } + if (PyDict_SetItem(self->function_pinboard, authorizer_cb, Py_None) == -1) { + return NULL; + } rc = sqlite3_set_authorizer(self->db, _authorizer_callback, (void*)authorizer_cb); - if (rc != SQLITE_OK) { PyErr_SetString(pysqlite_OperationalError, "Error setting authorizer callback"); return NULL; - } else { - if (PyDict_SetItem(self->function_pinboard, authorizer_cb, Py_None) == -1) - return NULL; - - Py_INCREF(Py_None); - return Py_None; } + Py_RETURN_NONE; } static PyObject* pysqlite_connection_set_progress_handler(pysqlite_Connection* self, PyObject* args, PyObject* kwargs) @@ -1041,9 +1034,9 @@ static PyObject* pysqlite_connection_set_progress_handler(pysqlite_Connection* s /* None clears the progress handler previously set */ sqlite3_progress_handler(self->db, 0, 0, (void*)0); } else { - sqlite3_progress_handler(self->db, n, _progress_handler, progress_handler); if (PyDict_SetItem(self->function_pinboard, progress_handler, Py_None) == -1) return NULL; + sqlite3_progress_handler(self->db, n, _progress_handler, progress_handler); } Py_INCREF(Py_None); @@ -1138,6 +1131,10 @@ static int pysqlite_connection_set_isolation_level(pysqlite_Connection* self, Py PyObject* begin_statement; char* begin_statement_str; + if (isolation_level == NULL) { + PyErr_SetString(PyExc_AttributeError, "cannot delete attribute"); + return -1; + } Py_XDECREF(self->isolation_level); if (self->begin_statement) { diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c index b7c2d79..e5cd721 100644 --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -147,7 +147,7 @@ PyObject* _pysqlite_get_converter(PyObject* key) return NULL; } - retval = PyDict_GetItem(converters, upcase_key); + retval = PyDict_GetItem(_pysqlite_converters, upcase_key); Py_DECREF(upcase_key); return retval; @@ -655,7 +655,7 @@ PyObject* _pysqlite_query_execute(pysqlite_Cursor* self, int multiple, PyObject* } else { if (PyErr_Occurred()) { /* there was an error that occurred in a user-defined callback */ - if (_enable_callback_tracebacks) { + if (_pysqlite_enable_callback_tracebacks) { PyErr_Print(); } else { PyErr_Clear(); diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index 33611d9..fa45edf 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -39,8 +39,8 @@ PyObject* pysqlite_Error, *pysqlite_Warning, *pysqlite_InterfaceError, *pysqlite *pysqlite_InternalError, *pysqlite_OperationalError, *pysqlite_ProgrammingError, *pysqlite_IntegrityError, *pysqlite_DataError, *pysqlite_NotSupportedError, *pysqlite_OptimizedUnicode; -PyObject* converters; -int _enable_callback_tracebacks; +PyObject* _pysqlite_converters; +int _pysqlite_enable_callback_tracebacks; int pysqlite_BaseTypeAdapted; static PyObject* module_connect(PyObject* self, PyObject* args, PyObject* @@ -190,7 +190,7 @@ static PyObject* module_register_converter(PyObject* self, PyObject* args) goto error; } - if (PyDict_SetItem(converters, name, callable) != 0) { + if (PyDict_SetItem(_pysqlite_converters, name, callable) != 0) { goto error; } @@ -208,7 +208,7 @@ Registers a converter with pysqlite. Non-standard."); static PyObject* enable_callback_tracebacks(PyObject* self, PyObject* args) { - if (!PyArg_ParseTuple(args, "i", &_enable_callback_tracebacks)) { + if (!PyArg_ParseTuple(args, "i", &_pysqlite_enable_callback_tracebacks)) { return NULL; } @@ -223,12 +223,12 @@ Enable or disable callback functions throwing errors to stderr."); static void converters_init(PyObject* dict) { - converters = PyDict_New(); - if (!converters) { + _pysqlite_converters = PyDict_New(); + if (!_pysqlite_converters) { return; } - PyDict_SetItemString(dict, "converters", converters); + PyDict_SetItemString(dict, "converters", _pysqlite_converters); } static PyMethodDef module_methods[] = { @@ -428,7 +428,7 @@ PyMODINIT_FUNC init_sqlite3(void) /* initialize the default converters */ converters_init(dict); - _enable_callback_tracebacks = 0; + _pysqlite_enable_callback_tracebacks = 0; pysqlite_BaseTypeAdapted = 0; diff --git a/Modules/_sqlite/module.h b/Modules/_sqlite/module.h index 8d17d61..322452f 100644 --- a/Modules/_sqlite/module.h +++ b/Modules/_sqlite/module.h @@ -40,17 +40,13 @@ extern PyObject* pysqlite_NotSupportedError; extern PyObject* pysqlite_OptimizedUnicode; -/* the functions time.time() and time.sleep() */ -extern PyObject* time_time; -extern PyObject* time_sleep; - /* A dictionary, mapping colum types (INTEGER, VARCHAR, etc.) to converter * functions, that convert the SQL value to the appropriate Python value. * The key is uppercase. */ -extern PyObject* converters; +extern PyObject* _pysqlite_converters; -extern int _enable_callback_tracebacks; +extern int _pysqlite_enable_callback_tracebacks; extern int pysqlite_BaseTypeAdapted; #define PARSE_DECLTYPES 1 diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c index 5d6263c..ec60095 100644 --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -205,7 +205,7 @@ static int _need_adapt(PyObject* obj) return 1; } - if (PyInt_CheckExact(obj) || PyLong_CheckExact(obj) + if (_PyAnyInt_CheckExact(obj) || PyFloat_CheckExact(obj) || PyString_CheckExact(obj) || PyUnicode_CheckExact(obj) || PyBuffer_Check(obj)) { return 0; diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c index a24dd8c..7377225 100644 --- a/Modules/_sqlite/util.c +++ b/Modules/_sqlite/util.c @@ -132,7 +132,7 @@ _pysqlite_long_from_int64(sqlite_int64 value) } # endif #endif - return PyInt_FromLong(value); + return PyInt_FromLong(Py_SAFE_DOWNCAST(value, sqlite_int64, long)); } sqlite_int64 diff --git a/Modules/_sre.c b/Modules/_sre.c index 94b3d50..081a1aa 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -835,7 +835,7 @@ entrance: if (ctx->pattern[0] == SRE_OP_INFO) { /* optimization info block */ /* <INFO> <1=skip> <2=flags> <3=min> ... */ - if (ctx->pattern[3] && (end - ctx->ptr) < ctx->pattern[3]) { + if (ctx->pattern[3] && (Py_uintptr_t)(end - ctx->ptr) < ctx->pattern[3]) { TRACE(("reject (got %" PY_FORMAT_SIZE_T "d chars, " "need %" PY_FORMAT_SIZE_T "d)\n", (end - ctx->ptr), (Py_ssize_t) ctx->pattern[3])); @@ -2882,7 +2882,7 @@ _compile(PyObject* self_, PyObject* args) skip = *code; \ VTRACE(("%lu (skip to %p)\n", \ (unsigned long)skip, code+skip)); \ - if (skip-adj > end-code) \ + if (skip-adj > (Py_uintptr_t)(end - code)) \ FAIL; \ code++; \ } while (0) @@ -2915,7 +2915,7 @@ _validate_charset(SRE_CODE *code, SRE_CODE *end) case SRE_OP_CHARSET: offset = 32/sizeof(SRE_CODE); /* 32-byte bitmap */ - if (offset > end-code) + if (offset > (Py_uintptr_t)(end - code)) FAIL; code += offset; break; @@ -2923,7 +2923,7 @@ _validate_charset(SRE_CODE *code, SRE_CODE *end) case SRE_OP_BIGCHARSET: GET_ARG; /* Number of blocks */ offset = 256/sizeof(SRE_CODE); /* 256-byte table */ - if (offset > end-code) + if (offset > (Py_uintptr_t)(end - code)) FAIL; /* Make sure that each byte points to a valid block */ for (i = 0; i < 256; i++) { @@ -2932,7 +2932,7 @@ _validate_charset(SRE_CODE *code, SRE_CODE *end) } code += offset; offset = arg * 32/sizeof(SRE_CODE); /* 32-byte bitmap times arg */ - if (offset > end-code) + if (offset > (Py_uintptr_t)(end - code)) FAIL; code += offset; break; @@ -2995,7 +2995,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) sre_match() code is robust even if they don't, and the worst you can get is nonsensical match results. */ GET_ARG; - if (arg > 2*groups+1) { + if (arg > 2 * (size_t)groups + 1) { VTRACE(("arg=%d, groups=%d\n", (int)arg, (int)groups)); FAIL; } @@ -3083,11 +3083,11 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) GET_ARG; prefix_len = arg; GET_ARG; /* prefix skip */ /* Here comes the prefix string */ - if (prefix_len > newcode-code) + if (prefix_len > (Py_uintptr_t)(newcode - code)) FAIL; code += prefix_len; /* And here comes the overlap table */ - if (prefix_len > newcode-code) + if (prefix_len > (Py_uintptr_t)(newcode - code)) FAIL; /* Each overlap value should be < prefix_len */ for (i = 0; i < prefix_len; i++) { @@ -3178,7 +3178,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) case SRE_OP_GROUPREF: case SRE_OP_GROUPREF_IGNORE: GET_ARG; - if (arg >= groups) + if (arg >= (size_t)groups) FAIL; break; @@ -3187,7 +3187,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) 'group' is either an integer group number or a group name, 'then' and 'else' are sub-regexes, and 'else' is optional. */ GET_ARG; - if (arg >= groups) + if (arg >= (size_t)groups) FAIL; GET_SKIP_ADJ(1); code--; /* The skip is relative to the first arg! */ @@ -3216,7 +3216,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) to allow arbitrary jumps anywhere in the code; so we just look for a JUMP opcode preceding our skip target. */ - if (skip >= 3 && skip-3 < end-code && + if (skip >= 3 && skip-3 < (Py_uintptr_t)(end - code) && code[skip-3] == SRE_OP_JUMP) { VTRACE(("both then and else parts present\n")); @@ -3327,7 +3327,7 @@ match_getindex(MatchObject* self, PyObject* index) { Py_ssize_t i; - if (PyInt_Check(index) || PyLong_Check(index)) + if (_PyAnyInt_Check(index)) return PyInt_AsSsize_t(index); i = -1; @@ -3335,7 +3335,7 @@ match_getindex(MatchObject* self, PyObject* index) if (self->pattern->groupindex) { index = PyObject_GetItem(self->pattern->groupindex, index); if (index) { - if (PyInt_Check(index) || PyLong_Check(index)) + if (_PyAnyInt_Check(index)) i = PyInt_AsSsize_t(index); Py_DECREF(index); } else diff --git a/Modules/_ssl.c b/Modules/_ssl.c index d0ce913..93b635c 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -74,6 +74,7 @@ #include "openssl/ssl.h" #include "openssl/err.h" #include "openssl/rand.h" +#include "openssl/dh.h" /* SSL error object */ static PyObject *PySSLErrorObject; @@ -1222,6 +1223,10 @@ _get_crl_dp(X509 *certificate) { STACK_OF(GENERAL_NAME) *gns; dp = sk_DIST_POINT_value(dps, i); + if (dp->distpoint == NULL) { + /* Ignore empty DP value, CVE-2019-5010 */ + continue; + } gns = dp->distpoint->name.fullname; for (j=0; j < sk_GENERAL_NAME_num(gns); j++) { @@ -2176,6 +2181,7 @@ context_new(PyTypeObject *type, PyObject *args, PyObject *kwds) int proto_version = PY_SSL_VERSION_TLS; long options; SSL_CTX *ctx = NULL; + int result; if (!PyArg_ParseTupleAndKeywords( args, kwds, "i:_SSLContext", kwlist, @@ -2240,8 +2246,38 @@ context_new(PyTypeObject *type, PyObject *args, PyObject *kwds) options |= SSL_OP_NO_SSLv2; if (proto_version != PY_SSL_VERSION_SSL3) options |= SSL_OP_NO_SSLv3; + /* Minimal security flags for server and client side context. + * Client sockets ignore server-side parameters. */ +#ifdef SSL_OP_NO_COMPRESSION + options |= SSL_OP_NO_COMPRESSION; +#endif +#ifdef SSL_OP_CIPHER_SERVER_PREFERENCE + options |= SSL_OP_CIPHER_SERVER_PREFERENCE; +#endif +#ifdef SSL_OP_SINGLE_DH_USE + options |= SSL_OP_SINGLE_DH_USE; +#endif +#ifdef SSL_OP_SINGLE_ECDH_USE + options |= SSL_OP_SINGLE_ECDH_USE; +#endif SSL_CTX_set_options(self->ctx, options); + /* A bare minimum cipher list without completly broken cipher suites. + * It's far from perfect but gives users a better head start. */ + if (proto_version != PY_SSL_VERSION_SSL2) { + result = SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL:!MD5"); + } else { + /* SSLv2 needs MD5 */ + result = SSL_CTX_set_cipher_list(ctx, "HIGH:!aNULL:!eNULL"); + } + if (result == 0) { + Py_DECREF(self); + ERR_clear_error(); + PyErr_SetString(PySSLErrorObject, + "No cipher can be selected."); + return NULL; + } + #if !defined(OPENSSL_NO_ECDH) && !defined(OPENSSL_VERSION_1_1) /* Allow automatic ECDH curve selection (on OpenSSL 1.0.2+), or use prime256v1 by default. This is Apache mod_ssl's initialization @@ -4375,6 +4411,10 @@ init_ssl(void) PyModule_AddIntConstant(m, "OP_NO_COMPRESSION", SSL_OP_NO_COMPRESSION); #endif +#ifdef SSL_OP_ENABLE_MIDDLEBOX_COMPAT + PyModule_AddIntConstant(m, "OP_ENABLE_MIDDLEBOX_COMPAT", + SSL_OP_ENABLE_MIDDLEBOX_COMPAT); +#endif #if HAVE_SNI r = Py_True; diff --git a/Modules/_struct.c b/Modules/_struct.c index d83d57f..8b97672 100644 --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -110,7 +110,7 @@ get_pylong(PyObject *v) PyObject *r, *w; int converted = 0; assert(v != NULL); - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!_PyAnyInt_Check(v)) { PyNumberMethods *m; /* Not an integer; first try to use __index__ to convert to an integer. If the __index__ method @@ -150,7 +150,7 @@ get_pylong(PyObject *v) v = m->nb_int(v); if (v == NULL) return NULL; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!_PyAnyInt_Check(v)) { PyErr_SetString(PyExc_TypeError, "__int__ method returned " "non-integer"); @@ -169,7 +169,7 @@ get_pylong(PyObject *v) /* Ensure we own a reference to v. */ Py_INCREF(v); - assert(PyInt_Check(v) || PyLong_Check(v)); + assert(_PyAnyInt_Check(v)); if (PyInt_Check(v)) { r = PyLong_FromLong(PyInt_AS_LONG(v)); Py_DECREF(v); diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 5902de0..67488e7 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1559,6 +1559,89 @@ getargs_et_hash(PyObject *self, PyObject *args) return result; } +static PyObject * +get_indices(PyObject *self, PyObject *args) +{ + int result; + PySliceObject *slice; + Py_ssize_t length, start, stop, step; + + if (!PyArg_ParseTuple(args, "On", &slice, &length)) + return NULL; + + result = PySlice_GetIndices(slice, length, &start, &stop, &step); + + if (PyErr_Occurred()) { + assert(result == -1); + return NULL; + } + + if (result == -1) { + Py_RETURN_NONE; + } + return Py_BuildValue("innn", result, start, stop, step); +} + +static PyObject * +parse_tuple_and_keywords(PyObject *self, PyObject *args) +{ + PyObject *sub_args; + PyObject *sub_kwargs; + const char *sub_format; + PyObject *sub_keywords; + + Py_ssize_t i, size; + char *keywords[8 + 1]; /* space for NULL at end */ + PyObject *o; + + int result; + PyObject *return_value = NULL; + + double buffers[8][4]; /* double ensures alignment where necessary */ + + if (!PyArg_ParseTuple(args, "OOsO:parse_tuple_and_keywords", + &sub_args, &sub_kwargs, + &sub_format, &sub_keywords)) + return NULL; + + if (!(PyList_CheckExact(sub_keywords) || PyTuple_CheckExact(sub_keywords))) { + PyErr_SetString(PyExc_ValueError, + "parse_tuple_and_keywords: sub_keywords must be either list or tuple"); + return NULL; + } + + memset(buffers, 0, sizeof(buffers)); + memset(keywords, 0, sizeof(keywords)); + + size = PySequence_Fast_GET_SIZE(sub_keywords); + if (size > 8) { + PyErr_SetString(PyExc_ValueError, + "parse_tuple_and_keywords: too many keywords in sub_keywords"); + goto exit; + } + + for (i = 0; i < size; i++) { + o = PySequence_Fast_GET_ITEM(sub_keywords, i); + keywords[i] = PyString_AsString(o); + if (keywords[i] == NULL) { + goto exit; + } + } + + result = PyArg_ParseTupleAndKeywords(sub_args, sub_kwargs, + sub_format, keywords, + buffers + 0, buffers + 1, buffers + 2, buffers + 3, + buffers + 4, buffers + 5, buffers + 6, buffers + 7); + + if (result) { + return_value = Py_None; + Py_INCREF(Py_None); + } + +exit: + return return_value; +} + #ifdef Py_USING_UNICODE static volatile int x; @@ -1963,7 +2046,8 @@ static int _pending_callback(void *arg) /* The following requests n callbacks to _pending_callback. It can be * run from any python thread. */ -PyObject *pending_threadfunc(PyObject *self, PyObject *arg) +static PyObject * +pending_threadfunc(PyObject *self, PyObject *arg) { PyObject *callable; int r; @@ -2603,6 +2687,8 @@ static PyMethodDef TestMethods[] = { #ifdef Py_USING_UNICODE {"test_empty_argparse", (PyCFunction)test_empty_argparse,METH_NOARGS}, #endif + {"get_indices", get_indices, METH_VARARGS}, + {"parse_tuple_and_keywords", parse_tuple_and_keywords, METH_VARARGS}, {"test_null_strings", (PyCFunction)test_null_strings, METH_NOARGS}, {"test_string_from_format", (PyCFunction)test_string_from_format, METH_NOARGS}, {"test_with_docstring", (PyCFunction)test_with_docstring, METH_NOARGS, diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 444c268..58640e2 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -1205,10 +1205,8 @@ AsObj(PyObject *value) } #endif - if(PyTclObject_Check(value)) { - Tcl_Obj *v = ((PyTclObject*)value)->value; - Tcl_IncrRefCount(v); - return v; + if (PyTclObject_Check(value)) { + return ((PyTclObject*)value)->value; } { @@ -2107,7 +2105,7 @@ Tkapp_GetInt(PyObject *self, PyObject *args) if (PyTuple_Size(args) == 1) { PyObject* o = PyTuple_GetItem(args, 0); - if (PyInt_Check(o) || PyLong_Check(o)) { + if (_PyAnyInt_Check(o)) { Py_INCREF(o); return o; } diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c index 5bd3a42..8d4eb09 100644 --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -1344,7 +1344,7 @@ array_fromlist(arrayobject *self, PyObject *list) Py_SIZE(self) += n; self->allocated = Py_SIZE(self); for (i = 0; i < n; i++) { - PyObject *v = PyList_GetItem(list, i); + PyObject *v = PyList_GET_ITEM(list, i); if ((*self->ob_descr->setitem)(self, Py_SIZE(self) - n + i, v) != 0) { Py_SIZE(self) -= n; @@ -1357,6 +1357,19 @@ array_fromlist(arrayobject *self, PyObject *list) self->allocated = Py_SIZE(self); return NULL; } + if (n != PyList_GET_SIZE(list)) { + PyErr_SetString(PyExc_RuntimeError, + "list changed size during iteration"); + Py_SIZE(self) -= n; + if (itemsize && (Py_SIZE(self) > PY_SSIZE_T_MAX / itemsize)) { + return PyErr_NoMemory(); + } + PyMem_RESIZE(item, char, + Py_SIZE(self) * itemsize); + self->ob_item = item; + self->allocated = Py_SIZE(self); + return NULL; + } } } Py_INCREF(Py_None); @@ -1383,7 +1396,7 @@ array_tolist(arrayobject *self, PyObject *unused) Py_DECREF(list); return NULL; } - PyList_SetItem(list, i, v); + PyList_SET_ITEM(list, i, v); } return list; } diff --git a/Modules/atexitmodule.c b/Modules/atexitmodule.c deleted file mode 100644 index 35ebf08..0000000 --- a/Modules/atexitmodule.c +++ /dev/null @@ -1,346 +0,0 @@ -/* - * atexit - allow programmer to define multiple exit functions to be executed - * upon normal program termination. - * - * Translated from atexit.py by Collin Winter. - + Copyright 2007 Python Software Foundation. - */ - -#include "Python.h" - -/* Forward declaration (for atexit_cleanup) */ -static PyObject *atexit_clear(PyObject*, PyObject*); -/* Forward declaration of module object */ -static struct PyModuleDef atexitmodule; - -/* ===================================================================== */ -/* Callback machinery. */ - -typedef struct { - PyObject *func; - PyObject *args; - PyObject *kwargs; -} atexit_callback; - -typedef struct { - atexit_callback **atexit_callbacks; - int ncallbacks; - int callback_len; -} atexitmodule_state; - -#define GET_ATEXIT_STATE(mod) ((atexitmodule_state*)PyModule_GetState(mod)) - - -static void -atexit_delete_cb(atexitmodule_state *modstate, int i) -{ - atexit_callback *cb; - - cb = modstate->atexit_callbacks[i]; - modstate->atexit_callbacks[i] = NULL; - Py_DECREF(cb->func); - Py_DECREF(cb->args); - Py_XDECREF(cb->kwargs); - PyMem_Free(cb); -} - -/* Clear all callbacks without calling them */ -static void -atexit_cleanup(atexitmodule_state *modstate) -{ - atexit_callback *cb; - int i; - for (i = 0; i < modstate->ncallbacks; i++) { - cb = modstate->atexit_callbacks[i]; - if (cb == NULL) - continue; - - atexit_delete_cb(modstate, i); - } - modstate->ncallbacks = 0; -} - -/* Installed into pylifecycle.c's atexit mechanism */ - -static void -atexit_callfuncs(void) -{ - PyObject *exc_type = NULL, *exc_value, *exc_tb, *r; - atexit_callback *cb; - PyObject *module; - atexitmodule_state *modstate; - int i; - - module = PyState_FindModule(&atexitmodule); - if (module == NULL) - return; - modstate = GET_ATEXIT_STATE(module); - - if (modstate->ncallbacks == 0) - return; - - - for (i = modstate->ncallbacks - 1; i >= 0; i--) - { - cb = modstate->atexit_callbacks[i]; - if (cb == NULL) - continue; - - r = PyObject_Call(cb->func, cb->args, cb->kwargs); - Py_XDECREF(r); - if (r == NULL) { - /* Maintain the last exception, but don't leak if there are - multiple exceptions. */ - if (exc_type) { - Py_DECREF(exc_type); - Py_XDECREF(exc_value); - Py_XDECREF(exc_tb); - } - PyErr_Fetch(&exc_type, &exc_value, &exc_tb); - if (!PyErr_GivenExceptionMatches(exc_type, PyExc_SystemExit)) { - PySys_WriteStderr("Error in atexit._run_exitfuncs:\n"); - PyErr_NormalizeException(&exc_type, &exc_value, &exc_tb); - PyErr_Display(exc_type, exc_value, exc_tb); - } - } - } - - atexit_cleanup(modstate); - - if (exc_type) - PyErr_Restore(exc_type, exc_value, exc_tb); -} - -/* ===================================================================== */ -/* Module methods. */ - -PyDoc_STRVAR(atexit_register__doc__, -"register(func, *args, **kwargs) -> func\n\ -\n\ -Register a function to be executed upon normal program termination\n\ -\n\ - func - function to be called at exit\n\ - args - optional arguments to pass to func\n\ - kwargs - optional keyword arguments to pass to func\n\ -\n\ - func is returned to facilitate usage as a decorator."); - -static PyObject * -atexit_register(PyObject *self, PyObject *args, PyObject *kwargs) -{ - atexitmodule_state *modstate; - atexit_callback *new_callback; - PyObject *func = NULL; - - modstate = GET_ATEXIT_STATE(self); - - if (modstate->ncallbacks >= modstate->callback_len) { - atexit_callback **r; - modstate->callback_len += 16; - r = (atexit_callback**)PyMem_Realloc(modstate->atexit_callbacks, - sizeof(atexit_callback*) * modstate->callback_len); - if (r == NULL) - return PyErr_NoMemory(); - modstate->atexit_callbacks = r; - } - - if (PyTuple_GET_SIZE(args) == 0) { - PyErr_SetString(PyExc_TypeError, - "register() takes at least 1 argument (0 given)"); - return NULL; - } - - func = PyTuple_GET_ITEM(args, 0); - if (!PyCallable_Check(func)) { - PyErr_SetString(PyExc_TypeError, - "the first argument must be callable"); - return NULL; - } - - new_callback = PyMem_Malloc(sizeof(atexit_callback)); - if (new_callback == NULL) - return PyErr_NoMemory(); - - new_callback->args = PyTuple_GetSlice(args, 1, PyTuple_GET_SIZE(args)); - if (new_callback->args == NULL) { - PyMem_Free(new_callback); - return NULL; - } - new_callback->func = func; - new_callback->kwargs = kwargs; - Py_INCREF(func); - Py_XINCREF(kwargs); - - modstate->atexit_callbacks[modstate->ncallbacks++] = new_callback; - - Py_INCREF(func); - return func; -} - -PyDoc_STRVAR(atexit_run_exitfuncs__doc__, -"_run_exitfuncs() -> None\n\ -\n\ -Run all registered exit functions."); - -static PyObject * -atexit_run_exitfuncs(PyObject *self, PyObject *unused) -{ - atexit_callfuncs(); - if (PyErr_Occurred()) - return NULL; - Py_RETURN_NONE; -} - -PyDoc_STRVAR(atexit_clear__doc__, -"_clear() -> None\n\ -\n\ -Clear the list of previously registered exit functions."); - -static PyObject * -atexit_clear(PyObject *self, PyObject *unused) -{ - atexit_cleanup(GET_ATEXIT_STATE(self)); - Py_RETURN_NONE; -} - -PyDoc_STRVAR(atexit_ncallbacks__doc__, -"_ncallbacks() -> int\n\ -\n\ -Return the number of registered exit functions."); - -static PyObject * -atexit_ncallbacks(PyObject *self, PyObject *unused) -{ - atexitmodule_state *modstate; - - modstate = GET_ATEXIT_STATE(self); - - return PyLong_FromSsize_t(modstate->ncallbacks); -} - -static int -atexit_m_traverse(PyObject *self, visitproc visit, void *arg) -{ - int i; - atexitmodule_state *modstate; - - modstate = GET_ATEXIT_STATE(self); - for (i = 0; i < modstate->ncallbacks; i++) { - atexit_callback *cb = modstate->atexit_callbacks[i]; - if (cb == NULL) - continue; - Py_VISIT(cb->func); - Py_VISIT(cb->args); - Py_VISIT(cb->kwargs); - } - return 0; -} - -static int -atexit_m_clear(PyObject *self) -{ - atexitmodule_state *modstate; - modstate = GET_ATEXIT_STATE(self); - atexit_cleanup(modstate); - return 0; -} - -static void -atexit_free(PyObject *m) -{ - atexitmodule_state *modstate; - modstate = GET_ATEXIT_STATE(m); - atexit_cleanup(modstate); - PyMem_Free(modstate->atexit_callbacks); -} - -PyDoc_STRVAR(atexit_unregister__doc__, -"unregister(func) -> None\n\ -\n\ -Unregister an exit function which was previously registered using\n\ -atexit.register\n\ -\n\ - func - function to be unregistered"); - -static PyObject * -atexit_unregister(PyObject *self, PyObject *func) -{ - atexitmodule_state *modstate; - atexit_callback *cb; - int i, eq; - - modstate = GET_ATEXIT_STATE(self); - - for (i = 0; i < modstate->ncallbacks; i++) - { - cb = modstate->atexit_callbacks[i]; - if (cb == NULL) - continue; - - eq = PyObject_RichCompareBool(cb->func, func, Py_EQ); - if (eq < 0) - return NULL; - if (eq) - atexit_delete_cb(modstate, i); - } - Py_RETURN_NONE; -} - -static PyMethodDef atexit_methods[] = { - {"register", (PyCFunction) atexit_register, METH_VARARGS|METH_KEYWORDS, - atexit_register__doc__}, - {"_clear", (PyCFunction) atexit_clear, METH_NOARGS, - atexit_clear__doc__}, - {"unregister", (PyCFunction) atexit_unregister, METH_O, - atexit_unregister__doc__}, - {"_run_exitfuncs", (PyCFunction) atexit_run_exitfuncs, METH_NOARGS, - atexit_run_exitfuncs__doc__}, - {"_ncallbacks", (PyCFunction) atexit_ncallbacks, METH_NOARGS, - atexit_ncallbacks__doc__}, - {NULL, NULL} /* sentinel */ -}; - -/* ===================================================================== */ -/* Initialization function. */ - -PyDoc_STRVAR(atexit__doc__, -"allow programmer to define multiple exit functions to be executed\ -upon normal program termination.\n\ -\n\ -Two public functions, register and unregister, are defined.\n\ -"); - - -static struct PyModuleDef atexitmodule = { - PyModuleDef_HEAD_INIT, - "atexit", - atexit__doc__, - sizeof(atexitmodule_state), - atexit_methods, - NULL, - atexit_m_traverse, - atexit_m_clear, - (freefunc)atexit_free -}; - -PyMODINIT_FUNC -PyInit_atexit(void) -{ - PyObject *m; - atexitmodule_state *modstate; - - m = PyModule_Create(&atexitmodule); - if (m == NULL) - return NULL; - - modstate = GET_ATEXIT_STATE(m); - modstate->callback_len = 32; - modstate->ncallbacks = 0; - modstate->atexit_callbacks = PyMem_New(atexit_callback*, - modstate->callback_len); - if (modstate->atexit_callbacks == NULL) - return NULL; - - _Py_PyAtExit(atexit_callfuncs); - return m; -} diff --git a/Modules/audioop.c b/Modules/audioop.c index ad7336c..91f585c 100644 --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -25,17 +25,25 @@ typedef short PyInt16; #endif static const int maxvals[] = {0, 0x7F, 0x7FFF, 0x7FFFFF, 0x7FFFFFFF}; -static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x80000000}; +/* -1 trick is needed on Windows to support -0x80000000 without a warning */ +static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x7FFFFFFF-1}; static const unsigned int masks[] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF}; static int fbound(double val, double minval, double maxval) { - if (val > maxval) + if (val > maxval) { val = maxval; - else if (val < minval + 1) + } + else if (val < minval + 1.0) { val = minval; - return val; + } + + /* Round towards minus infinity (-inf) */ + val = floor(val); + + /* Cast double to integer: round towards zero */ + return (int)val; } @@ -386,7 +394,9 @@ audioop_minmax(PyObject *self, PyObject *args) signed char *cp; int len, size, val = 0; int i; - int min = 0x7fffffff, max = -0x80000000; + /* -1 trick below is needed on Windows to support -0x80000000 without + a warning */ + int min = 0x7fffffff, max = -0x7FFFFFFF-1; if (!PyArg_ParseTuple(args, "s#i:minmax", &cp, &len, &size)) return NULL; @@ -793,8 +803,8 @@ audioop_mul(PyObject *self, PyObject *args) if ( size == 1 ) val = (int)*CHARP(cp, i); else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); - fval = (double)val*factor; - val = (int)floor(fbound(fval, minval, maxval)); + fval = (double)val * factor; + val = fbound(fval, minval, maxval); if ( size == 1 ) *CHARP(ncp, i) = (signed char)val; else if ( size == 2 ) *SHORTP(ncp, i) = (short)val; else if ( size == 4 ) *LONGP(ncp, i) = (Py_Int32)val; @@ -837,8 +847,8 @@ audioop_tomono(PyObject *self, PyObject *args) if ( size == 1 ) val2 = (int)*CHARP(cp, i+1); else if ( size == 2 ) val2 = (int)*SHORTP(cp, i+2); else if ( size == 4 ) val2 = (int)*LONGP(cp, i+4); - fval = (double)val1*fac1 + (double)val2*fac2; - val1 = (int)floor(fbound(fval, minval, maxval)); + fval = (double)val1 * fac1 + (double)val2 * fac2; + val1 = fbound(fval, minval, maxval); if ( size == 1 ) *CHARP(ncp, i/2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i/2) = (short)val1; else if ( size == 4 ) *LONGP(ncp, i/2)= (Py_Int32)val1; @@ -881,11 +891,11 @@ audioop_tostereo(PyObject *self, PyObject *args) else if ( size == 2 ) val = (int)*SHORTP(cp, i); else if ( size == 4 ) val = (int)*LONGP(cp, i); - fval = (double)val*fac1; - val1 = (int)floor(fbound(fval, minval, maxval)); + fval = (double)val * fac1; + val1 = fbound(fval, minval, maxval); - fval = (double)val*fac2; - val2 = (int)floor(fbound(fval, minval, maxval)); + fval = (double)val * fac2; + val2 = fbound(fval, minval, maxval); if ( size == 1 ) *CHARP(ncp, i*2) = (signed char)val1; else if ( size == 2 ) *SHORTP(ncp, i*2) = (short)val1; @@ -944,7 +954,7 @@ audioop_add(PyObject *self, PyObject *args) else { double fval = (double)val1 + (double)val2; /* truncate in case of overflow */ - newval = (int)floor(fbound(fval, minval, maxval)); + newval = fbound(fval, minval, maxval); } if ( size == 1 ) *CHARP(ncp, i) = (signed char)newval; diff --git a/Modules/cjkcodecs/_codecs_jp.c b/Modules/cjkcodecs/_codecs_jp.c index a05e01b..3acd19e 100644 --- a/Modules/cjkcodecs/_codecs_jp.c +++ b/Modules/cjkcodecs/_codecs_jp.c @@ -40,7 +40,7 @@ ENCODER(cp932) if (c == 0xf8f0) OUT1(0xa0) else - OUT1(c - 0xfef1 + 0xfd) + OUT1(c - 0xf8f1 + 0xfd) NEXT(1, 1) continue; } diff --git a/Modules/cjkcodecs/multibytecodec.c b/Modules/cjkcodecs/multibytecodec.c index 8901b42..1ad0ea0 100644 --- a/Modules/cjkcodecs/multibytecodec.c +++ b/Modules/cjkcodecs/multibytecodec.c @@ -138,6 +138,10 @@ codecctx_errors_set(MultibyteStatefulCodecContext *self, PyObject *value, { PyObject *cb; + if (value == NULL) { + PyErr_SetString(PyExc_AttributeError, "cannot delete attribute"); + return -1; + } if (!PyString_Check(value)) { PyErr_SetString(PyExc_TypeError, "errors must be a string"); return -1; @@ -310,8 +314,7 @@ multibytecodec_encerror(MultibyteCodec *codec, if (!PyTuple_Check(retobj) || PyTuple_GET_SIZE(retobj) != 2 || !PyUnicode_Check((tobj = PyTuple_GET_ITEM(retobj, 0))) || - !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) || - PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) { + !_PyAnyInt_Check(PyTuple_GET_ITEM(retobj, 1))) { PyErr_SetString(PyExc_TypeError, "encoding error handler must return " "(unicode, int) tuple"); @@ -430,8 +433,7 @@ multibytecodec_decerror(MultibyteCodec *codec, if (!PyTuple_Check(retobj) || PyTuple_GET_SIZE(retobj) != 2 || !PyUnicode_Check((retuni = PyTuple_GET_ITEM(retobj, 0))) || - !(PyInt_Check(PyTuple_GET_ITEM(retobj, 1)) || - PyLong_Check(PyTuple_GET_ITEM(retobj, 1)))) { + !_PyAnyInt_Check(PyTuple_GET_ITEM(retobj, 1))) { PyErr_SetString(PyExc_TypeError, "decoding error handler must return " "(unicode, int) tuple"); diff --git a/Modules/datetimemodule.c b/Modules/datetimemodule.c index e818c5c..29492de 100644 --- a/Modules/datetimemodule.c +++ b/Modules/datetimemodule.c @@ -1537,7 +1537,7 @@ delta_to_microseconds(PyDateTime_Delta *self) if (x2 == NULL) goto Done; result = PyNumber_Add(x1, x2); - assert(result == NULL || PyInt_CheckExact(result) || PyLong_CheckExact(result)); + assert(result == NULL || _PyAnyInt_CheckExact(result)); Done: Py_XDECREF(x1); @@ -1546,6 +1546,29 @@ Done: return result; } +static PyObject * +checked_divmod(PyObject *a, PyObject *b) +{ + PyObject *result = PyNumber_Divmod(a, b); + if (result != NULL) { + if (!PyTuple_Check(result)) { + PyErr_Format(PyExc_TypeError, + "divmod() returned non-tuple (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + if (PyTuple_GET_SIZE(result) != 2) { + PyErr_Format(PyExc_TypeError, + "divmod() returned a tuple of size %zd", + PyTuple_GET_SIZE(result)); + Py_DECREF(result); + return NULL; + } + } + return result; +} + /* Convert a number of us (as a Python int or long) to a timedelta. */ static PyObject * @@ -1554,70 +1577,49 @@ microseconds_to_delta_ex(PyObject *pyus, PyTypeObject *type) int us; int s; int d; - long temp; PyObject *tuple = NULL; PyObject *num = NULL; PyObject *result = NULL; - assert(PyInt_CheckExact(pyus) || PyLong_CheckExact(pyus)); - tuple = PyNumber_Divmod(pyus, us_per_second); - if (tuple == NULL) + tuple = checked_divmod(pyus, us_per_second); + if (tuple == NULL) { goto Done; + } - num = PyTuple_GetItem(tuple, 1); /* us */ - if (num == NULL) - goto Done; - temp = PyLong_AsLong(num); + num = PyTuple_GET_ITEM(tuple, 1); /* us */ + us = _PyLong_AsInt(num); num = NULL; - if (temp == -1 && PyErr_Occurred()) - goto Done; - assert(0 <= temp && temp < 1000000); - us = (int)temp; - if (us < 0) { - /* The divisor was positive, so this must be an error. */ - assert(PyErr_Occurred()); + if (us == -1 && PyErr_Occurred()) { goto Done; } + if (!(0 <= us && us < 1000000)) { + goto BadDivmod; + } - num = PyTuple_GetItem(tuple, 0); /* leftover seconds */ - if (num == NULL) - goto Done; + num = PyTuple_GET_ITEM(tuple, 0); /* leftover seconds */ Py_INCREF(num); Py_DECREF(tuple); - tuple = PyNumber_Divmod(num, seconds_per_day); + tuple = checked_divmod(num, seconds_per_day); if (tuple == NULL) goto Done; Py_DECREF(num); - num = PyTuple_GetItem(tuple, 1); /* seconds */ - if (num == NULL) - goto Done; - temp = PyLong_AsLong(num); + num = PyTuple_GET_ITEM(tuple, 1); /* seconds */ + s = _PyLong_AsInt(num); num = NULL; - if (temp == -1 && PyErr_Occurred()) - goto Done; - assert(0 <= temp && temp < 24*3600); - s = (int)temp; - - if (s < 0) { - /* The divisor was positive, so this must be an error. */ - assert(PyErr_Occurred()); + if (s == -1 && PyErr_Occurred()) { goto Done; } + if (!(0 <= s && s < 24*3600)) { + goto BadDivmod; + } - num = PyTuple_GetItem(tuple, 0); /* leftover days */ - if (num == NULL) - goto Done; + num = PyTuple_GET_ITEM(tuple, 0); /* leftover days */ Py_INCREF(num); - temp = PyLong_AsLong(num); - if (temp == -1 && PyErr_Occurred()) - goto Done; - d = (int)temp; - if ((long)d != temp) { - PyErr_SetString(PyExc_OverflowError, "normalized days too " - "large to fit in a C int"); + d = _PyLong_AsInt(num); + if (d == -1 && PyErr_Occurred()) { goto Done; } result = new_delta_ex(d, s, us, 0, type); @@ -1626,6 +1628,11 @@ Done: Py_XDECREF(tuple); Py_XDECREF(num); return result; + +BadDivmod: + PyErr_SetString(PyExc_TypeError, + "divmod() returned a value out of range"); + goto Done; } #define microseconds_to_delta(pymicros) \ @@ -1642,7 +1649,7 @@ multiply_int_timedelta(PyObject *intobj, PyDateTime_Delta *delta) if (pyus_in == NULL) return NULL; - pyus_out = PyNumber_Multiply(pyus_in, intobj); + pyus_out = PyNumber_Multiply(intobj, pyus_in); Py_DECREF(pyus_in); if (pyus_out == NULL) return NULL; @@ -1803,11 +1810,11 @@ delta_multiply(PyObject *left, PyObject *right) if (PyDelta_Check(left)) { /* delta * ??? */ - if (PyInt_Check(right) || PyLong_Check(right)) + if (_PyAnyInt_Check(right)) result = multiply_int_timedelta(right, (PyDateTime_Delta *) left); } - else if (PyInt_Check(left) || PyLong_Check(left)) + else if (_PyAnyInt_Check(left)) result = multiply_int_timedelta(left, (PyDateTime_Delta *) right); @@ -1823,7 +1830,7 @@ delta_divide(PyObject *left, PyObject *right) if (PyDelta_Check(left)) { /* delta * ??? */ - if (PyInt_Check(right) || PyLong_Check(right)) + if (_PyAnyInt_Check(right)) result = divide_timedelta_int( (PyDateTime_Delta *)left, right); @@ -1852,14 +1859,12 @@ accum(const char* tag, PyObject *sofar, PyObject *num, PyObject *factor, assert(num != NULL); - if (PyInt_Check(num) || PyLong_Check(num)) { - prod = PyNumber_Multiply(factor, num); + if (_PyAnyInt_Check(num)) { + prod = PyNumber_Multiply(num, factor); if (prod == NULL) return NULL; - assert(PyInt_CheckExact(prod) || PyLong_CheckExact(prod)); sum = PyNumber_Add(sofar, prod); Py_DECREF(prod); - assert(sum == NULL || PyInt_CheckExact(sum) || PyLong_CheckExact(sum)); return sum; } @@ -1902,7 +1907,7 @@ accum(const char* tag, PyObject *sofar, PyObject *num, PyObject *factor, * fractional part requires float arithmetic, and may * lose a little info. */ - assert(PyInt_CheckExact(factor) || PyLong_CheckExact(factor)); + assert(_PyAnyInt_CheckExact(factor)); if (PyInt_Check(factor)) dnum = (double)PyInt_AsLong(factor); else @@ -1920,7 +1925,6 @@ accum(const char* tag, PyObject *sofar, PyObject *num, PyObject *factor, Py_DECREF(sum); Py_DECREF(x); *leftover += fracpart; - assert(y == NULL || PyInt_CheckExact(y) || PyLong_CheckExact(y)); return y; } diff --git a/Modules/dlmodule.c b/Modules/dlmodule.c index 7a6686e..3f15048 100644 --- a/Modules/dlmodule.c +++ b/Modules/dlmodule.c @@ -107,7 +107,7 @@ dl_call(dlobject *xp, PyObject *args) } for (i = 1; i < n; i++) { PyObject *v = PyTuple_GetItem(args, i); - if (PyInt_Check(v) || PyLong_Check(v)) { + if (_PyAnyInt_Check(v)) { alist[i-1] = PyInt_AsLong(v); if (alist[i-1] == -1 && PyErr_Occurred()) return NULL; diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h index d0735bb..174c3fa 100644 --- a/Modules/expat/expat.h +++ b/Modules/expat/expat.h @@ -264,7 +264,7 @@ XML_ParserCreate_MM(const XML_Char *encoding, const XML_Char *namespaceSeparator); /* Prepare a parser object to be re-used. This is particularly - valuable when memory allocation overhead is disproportionatly high, + valuable when memory allocation overhead is disproportionately high, such as when a large number of small documnents need to be parsed. All handlers are cleared from the parser, except for the unknownEncodingHandler. The parser's external state is re-initialized @@ -1076,7 +1076,7 @@ XML_GetFeatureList(void); */ #define XML_MAJOR_VERSION 2 #define XML_MINOR_VERSION 2 -#define XML_MICRO_VERSION 4 +#define XML_MICRO_VERSION 6 #ifdef __cplusplus } diff --git a/Modules/expat/expat_external.h b/Modules/expat/expat_external.h index 8110285..629483a 100644 --- a/Modules/expat/expat_external.h +++ b/Modules/expat/expat_external.h @@ -35,12 +35,8 @@ /* External API definitions */ -/* Namespace external symbols to allow multiple libexpat version to - co-exist. */ -#include "pyexpatns.h" - #if defined(_MSC_EXTENSIONS) && !defined(__BEOS__) && !defined(__CYGWIN__) -#define XML_USE_MSC_EXTENSIONS 1 +# define XML_USE_MSC_EXTENSIONS 1 #endif /* Expat tries very hard to make the API boundary very specifically @@ -66,11 +62,11 @@ system headers may assume the cdecl convention. */ #ifndef XMLCALL -#if defined(_MSC_VER) -#define XMLCALL __cdecl -#elif defined(__GNUC__) && defined(__i386) && !defined(__INTEL_COMPILER) -#define XMLCALL __attribute__((cdecl)) -#else +# if defined(_MSC_VER) +# define XMLCALL __cdecl +# elif defined(__GNUC__) && defined(__i386) && !defined(__INTEL_COMPILER) +# define XMLCALL __attribute__((cdecl)) +# else /* For any platform which uses this definition and supports more than one calling convention, we need to extend this definition to declare the convention used on that platform, if it's possible to @@ -81,41 +77,41 @@ pre-processor and how to specify the same calling convention as the platform's malloc() implementation. */ -#define XMLCALL -#endif +# define XMLCALL +# endif #endif /* not defined XMLCALL */ #if !defined(XML_STATIC) && !defined(XMLIMPORT) -#ifndef XML_BUILDING_EXPAT +# ifndef XML_BUILDING_EXPAT /* using Expat from an application */ -#ifdef XML_USE_MSC_EXTENSIONS -#define XMLIMPORT __declspec(dllimport) -#endif +# ifdef XML_USE_MSC_EXTENSIONS +# define XMLIMPORT __declspec(dllimport) +# endif -#endif +# endif #endif /* not defined XML_STATIC */ #if !defined(XMLIMPORT) && defined(__GNUC__) && (__GNUC__ >= 4) -#define XMLIMPORT __attribute__ ((visibility ("default"))) +# define XMLIMPORT __attribute__ ((visibility ("default"))) #endif /* If we didn't define it above, define it away: */ #ifndef XMLIMPORT -#define XMLIMPORT +# define XMLIMPORT #endif #if defined(__GNUC__) && (__GNUC__ > 2 || (__GNUC__ == 2 && __GNUC_MINOR__ >= 96)) -#define XML_ATTR_MALLOC __attribute__((__malloc__)) +# define XML_ATTR_MALLOC __attribute__((__malloc__)) #else -#define XML_ATTR_MALLOC +# define XML_ATTR_MALLOC #endif #if defined(__GNUC__) && ((__GNUC__ > 4) || (__GNUC__ == 4 && __GNUC_MINOR__ >= 3)) -#define XML_ATTR_ALLOC_SIZE(x) __attribute__((__alloc_size__(x))) +# define XML_ATTR_ALLOC_SIZE(x) __attribute__((__alloc_size__(x))) #else -#define XML_ATTR_ALLOC_SIZE(x) +# define XML_ATTR_ALLOC_SIZE(x) #endif #define XMLPARSEAPI(type) XMLIMPORT type XMLCALL @@ -125,33 +121,35 @@ extern "C" { #endif #ifdef XML_UNICODE_WCHAR_T -# define XML_UNICODE +# ifndef XML_UNICODE +# define XML_UNICODE +# endif # if defined(__SIZEOF_WCHAR_T__) && (__SIZEOF_WCHAR_T__ != 2) # error "sizeof(wchar_t) != 2; Need -fshort-wchar for both Expat and libc" # endif #endif #ifdef XML_UNICODE /* Information is UTF-16 encoded. */ -#ifdef XML_UNICODE_WCHAR_T +# ifdef XML_UNICODE_WCHAR_T typedef wchar_t XML_Char; typedef wchar_t XML_LChar; -#else +# else typedef unsigned short XML_Char; typedef char XML_LChar; -#endif /* XML_UNICODE_WCHAR_T */ +# endif /* XML_UNICODE_WCHAR_T */ #else /* Information is UTF-8 encoded. */ typedef char XML_Char; typedef char XML_LChar; #endif /* XML_UNICODE */ #ifdef XML_LARGE_SIZE /* Use large integers for file/stream positions. */ -#if defined(XML_USE_MSC_EXTENSIONS) && _MSC_VER < 1400 +# if defined(XML_USE_MSC_EXTENSIONS) && _MSC_VER < 1400 typedef __int64 XML_Index; typedef unsigned __int64 XML_Size; -#else +# else typedef long long XML_Index; typedef unsigned long long XML_Size; -#endif +# endif #else typedef long XML_Index; typedef unsigned long XML_Size; diff --git a/Modules/expat/internal.h b/Modules/expat/internal.h index 3c5d6e9..e33fdcb 100644 --- a/Modules/expat/internal.h +++ b/Modules/expat/internal.h @@ -116,7 +116,7 @@ extern "C" { void -align_limit_to_full_utf8_characters(const char * from, const char ** fromLimRef); +_INTERNAL_trim_to_complete_utf8_characters(const char * from, const char ** fromLimRef); #ifdef __cplusplus diff --git a/Modules/expat/loadlibrary.c b/Modules/expat/loadlibrary.c index 452ae92..35fdf98 100644 --- a/Modules/expat/loadlibrary.c +++ b/Modules/expat/loadlibrary.c @@ -84,7 +84,7 @@ HMODULE _Expat_LoadLibrary(LPCTSTR filename) /* Get a handle to kernel32 so we can access it's functions at runtime */ HMODULE hKernel32 = GetModuleHandle(TEXT("kernel32")); if(!hKernel32) - return NULL; + return NULL; /* LCOV_EXCL_LINE */ /* Attempt to find LoadLibraryEx() which is only available on Windows 2000 and above */ diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h index 581872d..4d6786d 100644 --- a/Modules/expat/siphash.h +++ b/Modules/expat/siphash.h @@ -11,6 +11,12 @@ * -------------------------------------------------------------------------- * HISTORY: * + * 2018-07-08 (Anton Maklakov) + * - Add "fall through" markers for GCC's -Wimplicit-fallthrough + * + * 2017-11-03 (Sebastian Pipping) + * - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined + * * 2017-07-25 (Vadim Zeitlin) * - Fix use of SIPHASH_MAIN macro * @@ -151,6 +157,8 @@ static struct sipkey *sip_tokey(struct sipkey *key, const void *src) { } /* sip_tokey() */ +#ifdef SIPHASH_TOBIN + #define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v)) static void *sip_tobin(void *dst, uint64_t u64) { @@ -158,6 +166,8 @@ static void *sip_tobin(void *dst, uint64_t u64) { return dst; } /* sip_tobin() */ +#endif /* SIPHASH_TOBIN */ + static void sip_round(struct siphash *H, const int rounds) { int i; @@ -231,12 +241,19 @@ static uint64_t sip24_final(struct siphash *H) { switch (left) { case 7: b |= (uint64_t)H->buf[6] << 48; + /* fall through */ case 6: b |= (uint64_t)H->buf[5] << 40; + /* fall through */ case 5: b |= (uint64_t)H->buf[4] << 32; + /* fall through */ case 4: b |= (uint64_t)H->buf[3] << 24; + /* fall through */ case 3: b |= (uint64_t)H->buf[2] << 16; + /* fall through */ case 2: b |= (uint64_t)H->buf[1] << 8; + /* fall through */ case 1: b |= (uint64_t)H->buf[0] << 0; + /* fall through */ case 0: break; } diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 0df6883..2a76345 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -1,4 +1,4 @@ -/* 8c6b2be7c6281da65ce05218fc15c339f02a811706340824ab596aa86e1fd51a (2.2.4+) +/* 19ac4776051591216f1874e34ee99b6a43a3784c8bd7d70efeb9258dd22b906a (2.2.6+) __ __ _ ___\ \/ /_ __ __ _| |_ / _ \\ /| '_ \ / _` | __| @@ -30,6 +30,9 @@ USE OR OTHER DEALINGS IN THE SOFTWARE. */ +#if !defined(_WIN32) && defined(HAVE_EXPAT_CONFIG_H) +# include <pyconfig.h> +#endif #if !defined(_GNU_SOURCE) # define _GNU_SOURCE 1 /* syscall prototype */ #endif @@ -161,6 +164,9 @@ typedef char ICHAR; /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) +/* Do safe (NULL-aware) pointer arithmetic */ +#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0) + /* Handle the case where memmove() doesn't exist. */ #ifndef HAVE_MEMMOVE #ifdef HAVE_BCOPY @@ -470,7 +476,7 @@ setContext(XML_Parser parser, const XML_Char *context); static void FASTCALL normalizePublicId(XML_Char *s); static DTD * dtdCreate(const XML_Memory_Handling_Suite *ms); -/* do not call if parentParser != NULL */ +/* do not call if m_parentParser != NULL */ static void dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms); static void dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms); @@ -542,7 +548,7 @@ parserInit(XML_Parser parser, const XML_Char *encodingName); : ((*((pool)->ptr)++ = c), 1)) struct XML_ParserStruct { - /* The first member must be userData so that the XML_GetUserData + /* The first member must be m_userData so that the XML_GetUserData macro works. */ void *m_userData; void *m_handlerArg; @@ -552,7 +558,7 @@ struct XML_ParserStruct { const char *m_bufferPtr; /* past last character to be parsed */ char *m_bufferEnd; - /* allocated end of buffer */ + /* allocated end of m_buffer */ const char *m_bufferLim; XML_Index m_parseEndByteIndex; const char *m_parseEndPtr; @@ -644,113 +650,10 @@ struct XML_ParserStruct { unsigned long m_hash_secret_salt; }; -#define MALLOC(s) (parser->m_mem.malloc_fcn((s))) -#define REALLOC(p,s) (parser->m_mem.realloc_fcn((p),(s))) -#define FREE(p) (parser->m_mem.free_fcn((p))) - -#define userData (parser->m_userData) -#define handlerArg (parser->m_handlerArg) -#define startElementHandler (parser->m_startElementHandler) -#define endElementHandler (parser->m_endElementHandler) -#define characterDataHandler (parser->m_characterDataHandler) -#define processingInstructionHandler \ - (parser->m_processingInstructionHandler) -#define commentHandler (parser->m_commentHandler) -#define startCdataSectionHandler \ - (parser->m_startCdataSectionHandler) -#define endCdataSectionHandler (parser->m_endCdataSectionHandler) -#define defaultHandler (parser->m_defaultHandler) -#define startDoctypeDeclHandler (parser->m_startDoctypeDeclHandler) -#define endDoctypeDeclHandler (parser->m_endDoctypeDeclHandler) -#define unparsedEntityDeclHandler \ - (parser->m_unparsedEntityDeclHandler) -#define notationDeclHandler (parser->m_notationDeclHandler) -#define startNamespaceDeclHandler \ - (parser->m_startNamespaceDeclHandler) -#define endNamespaceDeclHandler (parser->m_endNamespaceDeclHandler) -#define notStandaloneHandler (parser->m_notStandaloneHandler) -#define externalEntityRefHandler \ - (parser->m_externalEntityRefHandler) -#define externalEntityRefHandlerArg \ - (parser->m_externalEntityRefHandlerArg) -#define internalEntityRefHandler \ - (parser->m_internalEntityRefHandler) -#define skippedEntityHandler (parser->m_skippedEntityHandler) -#define unknownEncodingHandler (parser->m_unknownEncodingHandler) -#define elementDeclHandler (parser->m_elementDeclHandler) -#define attlistDeclHandler (parser->m_attlistDeclHandler) -#define entityDeclHandler (parser->m_entityDeclHandler) -#define xmlDeclHandler (parser->m_xmlDeclHandler) -#define encoding (parser->m_encoding) -#define initEncoding (parser->m_initEncoding) -#define internalEncoding (parser->m_internalEncoding) -#define unknownEncodingMem (parser->m_unknownEncodingMem) -#define unknownEncodingData (parser->m_unknownEncodingData) -#define unknownEncodingHandlerData \ - (parser->m_unknownEncodingHandlerData) -#define unknownEncodingRelease (parser->m_unknownEncodingRelease) -#define protocolEncodingName (parser->m_protocolEncodingName) -#define ns (parser->m_ns) -#define ns_triplets (parser->m_ns_triplets) -#define prologState (parser->m_prologState) -#define processor (parser->m_processor) -#define errorCode (parser->m_errorCode) -#define eventPtr (parser->m_eventPtr) -#define eventEndPtr (parser->m_eventEndPtr) -#define positionPtr (parser->m_positionPtr) -#define position (parser->m_position) -#define openInternalEntities (parser->m_openInternalEntities) -#define freeInternalEntities (parser->m_freeInternalEntities) -#define defaultExpandInternalEntities \ - (parser->m_defaultExpandInternalEntities) -#define tagLevel (parser->m_tagLevel) -#define buffer (parser->m_buffer) -#define bufferPtr (parser->m_bufferPtr) -#define bufferEnd (parser->m_bufferEnd) -#define parseEndByteIndex (parser->m_parseEndByteIndex) -#define parseEndPtr (parser->m_parseEndPtr) -#define bufferLim (parser->m_bufferLim) -#define dataBuf (parser->m_dataBuf) -#define dataBufEnd (parser->m_dataBufEnd) -#define _dtd (parser->m_dtd) -#define curBase (parser->m_curBase) -#define declEntity (parser->m_declEntity) -#define doctypeName (parser->m_doctypeName) -#define doctypeSysid (parser->m_doctypeSysid) -#define doctypePubid (parser->m_doctypePubid) -#define declAttributeType (parser->m_declAttributeType) -#define declNotationName (parser->m_declNotationName) -#define declNotationPublicId (parser->m_declNotationPublicId) -#define declElementType (parser->m_declElementType) -#define declAttributeId (parser->m_declAttributeId) -#define declAttributeIsCdata (parser->m_declAttributeIsCdata) -#define declAttributeIsId (parser->m_declAttributeIsId) -#define freeTagList (parser->m_freeTagList) -#define freeBindingList (parser->m_freeBindingList) -#define inheritedBindings (parser->m_inheritedBindings) -#define tagStack (parser->m_tagStack) -#define atts (parser->m_atts) -#define attsSize (parser->m_attsSize) -#define nSpecifiedAtts (parser->m_nSpecifiedAtts) -#define idAttIndex (parser->m_idAttIndex) -#define nsAtts (parser->m_nsAtts) -#define nsAttsVersion (parser->m_nsAttsVersion) -#define nsAttsPower (parser->m_nsAttsPower) -#define attInfo (parser->m_attInfo) -#define tempPool (parser->m_tempPool) -#define temp2Pool (parser->m_temp2Pool) -#define groupConnector (parser->m_groupConnector) -#define groupSize (parser->m_groupSize) -#define namespaceSeparator (parser->m_namespaceSeparator) -#define parentParser (parser->m_parentParser) -#define ps_parsing (parser->m_parsingStatus.parsing) -#define ps_finalBuffer (parser->m_parsingStatus.finalBuffer) -#ifdef XML_DTD -#define isParamEntity (parser->m_isParamEntity) -#define useForeignDTD (parser->m_useForeignDTD) -#define paramEntityParsing (parser->m_paramEntityParsing) -#endif /* XML_DTD */ -#define hash_secret_salt (parser->m_hash_secret_salt) +#define MALLOC(parser, s) (parser->m_mem.malloc_fcn((s))) +#define REALLOC(parser, p, s) (parser->m_mem.realloc_fcn((p),(s))) +#define FREE(parser, p) (parser->m_mem.free_fcn((p))) + XML_Parser XMLCALL XML_ParserCreate(const XML_Char *encodingName) @@ -776,6 +679,9 @@ static const XML_Char implicitContext[] = { }; +/* To avoid warnings about unused functions: */ +#if ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM) + #if defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM) /* Obtain entropy on Linux 3.17+ */ @@ -841,6 +747,8 @@ writeRandomBytes_dev_urandom(void * target, size_t count) { #endif /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */ +#endif /* ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM) */ + #if defined(HAVE_ARC4RANDOM) @@ -942,6 +850,8 @@ generate_hash_secret_salt(XML_Parser parser) { unsigned long entropy; (void)parser; + + /* "Failproof" high quality providers: */ #if defined(HAVE_ARC4RANDOM_BUF) arc4random_buf(&entropy, sizeof(entropy)); return ENTROPY_DEBUG("arc4random_buf", entropy); @@ -990,9 +900,9 @@ static XML_Bool /* only valid for root parser */ startParsing(XML_Parser parser) { /* hash functions must be initialized before setContext() is called */ - if (hash_secret_salt == 0) - hash_secret_salt = generate_hash_secret_salt(parser); - if (ns) { + if (parser->m_hash_secret_salt == 0) + parser->m_hash_secret_salt = generate_hash_secret_salt(parser); + if (parser->m_ns) { /* implicit context only set for root parser, since child parsers (i.e. external entity parsers) will inherit it */ @@ -1042,85 +952,85 @@ parserCreate(const XML_Char *encodingName, if (!parser) return parser; - buffer = NULL; - bufferLim = NULL; + parser->m_buffer = NULL; + parser->m_bufferLim = NULL; - attsSize = INIT_ATTS_SIZE; - atts = (ATTRIBUTE *)MALLOC(attsSize * sizeof(ATTRIBUTE)); - if (atts == NULL) { - FREE(parser); + parser->m_attsSize = INIT_ATTS_SIZE; + parser->m_atts = (ATTRIBUTE *)MALLOC(parser, parser->m_attsSize * sizeof(ATTRIBUTE)); + if (parser->m_atts == NULL) { + FREE(parser, parser); return NULL; } #ifdef XML_ATTR_INFO - attInfo = (XML_AttrInfo*)MALLOC(attsSize * sizeof(XML_AttrInfo)); - if (attInfo == NULL) { - FREE(atts); - FREE(parser); + parser->m_attInfo = (XML_AttrInfo*)MALLOC(parser, parser->m_attsSize * sizeof(XML_AttrInfo)); + if (parser->m_attInfo == NULL) { + FREE(parser, parser->m_atts); + FREE(parser, parser); return NULL; } #endif - dataBuf = (XML_Char *)MALLOC(INIT_DATA_BUF_SIZE * sizeof(XML_Char)); - if (dataBuf == NULL) { - FREE(atts); + parser->m_dataBuf = (XML_Char *)MALLOC(parser, INIT_DATA_BUF_SIZE * sizeof(XML_Char)); + if (parser->m_dataBuf == NULL) { + FREE(parser, parser->m_atts); #ifdef XML_ATTR_INFO - FREE(attInfo); + FREE(parser, parser->m_attInfo); #endif - FREE(parser); + FREE(parser, parser); return NULL; } - dataBufEnd = dataBuf + INIT_DATA_BUF_SIZE; + parser->m_dataBufEnd = parser->m_dataBuf + INIT_DATA_BUF_SIZE; if (dtd) - _dtd = dtd; + parser->m_dtd = dtd; else { - _dtd = dtdCreate(&parser->m_mem); - if (_dtd == NULL) { - FREE(dataBuf); - FREE(atts); + parser->m_dtd = dtdCreate(&parser->m_mem); + if (parser->m_dtd == NULL) { + FREE(parser, parser->m_dataBuf); + FREE(parser, parser->m_atts); #ifdef XML_ATTR_INFO - FREE(attInfo); + FREE(parser, parser->m_attInfo); #endif - FREE(parser); + FREE(parser, parser); return NULL; } } - freeBindingList = NULL; - freeTagList = NULL; - freeInternalEntities = NULL; + parser->m_freeBindingList = NULL; + parser->m_freeTagList = NULL; + parser->m_freeInternalEntities = NULL; - groupSize = 0; - groupConnector = NULL; + parser->m_groupSize = 0; + parser->m_groupConnector = NULL; - unknownEncodingHandler = NULL; - unknownEncodingHandlerData = NULL; + parser->m_unknownEncodingHandler = NULL; + parser->m_unknownEncodingHandlerData = NULL; - namespaceSeparator = ASCII_EXCL; - ns = XML_FALSE; - ns_triplets = XML_FALSE; + parser->m_namespaceSeparator = ASCII_EXCL; + parser->m_ns = XML_FALSE; + parser->m_ns_triplets = XML_FALSE; - nsAtts = NULL; - nsAttsVersion = 0; - nsAttsPower = 0; + parser->m_nsAtts = NULL; + parser->m_nsAttsVersion = 0; + parser->m_nsAttsPower = 0; - protocolEncodingName = NULL; + parser->m_protocolEncodingName = NULL; - poolInit(&tempPool, &(parser->m_mem)); - poolInit(&temp2Pool, &(parser->m_mem)); + poolInit(&parser->m_tempPool, &(parser->m_mem)); + poolInit(&parser->m_temp2Pool, &(parser->m_mem)); parserInit(parser, encodingName); - if (encodingName && !protocolEncodingName) { + if (encodingName && !parser->m_protocolEncodingName) { XML_ParserFree(parser); return NULL; } if (nameSep) { - ns = XML_TRUE; - internalEncoding = XmlGetInternalEncodingNS(); - namespaceSeparator = *nameSep; + parser->m_ns = XML_TRUE; + parser->m_internalEncoding = XmlGetInternalEncodingNS(); + parser->m_namespaceSeparator = *nameSep; } else { - internalEncoding = XmlGetInternalEncoding(); + parser->m_internalEncoding = XmlGetInternalEncoding(); } return parser; @@ -1129,85 +1039,85 @@ parserCreate(const XML_Char *encodingName, static void parserInit(XML_Parser parser, const XML_Char *encodingName) { - processor = prologInitProcessor; - XmlPrologStateInit(&prologState); + parser->m_processor = prologInitProcessor; + XmlPrologStateInit(&parser->m_prologState); if (encodingName != NULL) { - protocolEncodingName = copyString(encodingName, &(parser->m_mem)); - } - curBase = NULL; - XmlInitEncoding(&initEncoding, &encoding, 0); - userData = NULL; - handlerArg = NULL; - startElementHandler = NULL; - endElementHandler = NULL; - characterDataHandler = NULL; - processingInstructionHandler = NULL; - commentHandler = NULL; - startCdataSectionHandler = NULL; - endCdataSectionHandler = NULL; - defaultHandler = NULL; - startDoctypeDeclHandler = NULL; - endDoctypeDeclHandler = NULL; - unparsedEntityDeclHandler = NULL; - notationDeclHandler = NULL; - startNamespaceDeclHandler = NULL; - endNamespaceDeclHandler = NULL; - notStandaloneHandler = NULL; - externalEntityRefHandler = NULL; - externalEntityRefHandlerArg = parser; - skippedEntityHandler = NULL; - elementDeclHandler = NULL; - attlistDeclHandler = NULL; - entityDeclHandler = NULL; - xmlDeclHandler = NULL; - bufferPtr = buffer; - bufferEnd = buffer; - parseEndByteIndex = 0; - parseEndPtr = NULL; - declElementType = NULL; - declAttributeId = NULL; - declEntity = NULL; - doctypeName = NULL; - doctypeSysid = NULL; - doctypePubid = NULL; - declAttributeType = NULL; - declNotationName = NULL; - declNotationPublicId = NULL; - declAttributeIsCdata = XML_FALSE; - declAttributeIsId = XML_FALSE; - memset(&position, 0, sizeof(POSITION)); - errorCode = XML_ERROR_NONE; - eventPtr = NULL; - eventEndPtr = NULL; - positionPtr = NULL; - openInternalEntities = NULL; - defaultExpandInternalEntities = XML_TRUE; - tagLevel = 0; - tagStack = NULL; - inheritedBindings = NULL; - nSpecifiedAtts = 0; - unknownEncodingMem = NULL; - unknownEncodingRelease = NULL; - unknownEncodingData = NULL; - parentParser = NULL; - ps_parsing = XML_INITIALIZED; + parser->m_protocolEncodingName = copyString(encodingName, &(parser->m_mem)); + } + parser->m_curBase = NULL; + XmlInitEncoding(&parser->m_initEncoding, &parser->m_encoding, 0); + parser->m_userData = NULL; + parser->m_handlerArg = NULL; + parser->m_startElementHandler = NULL; + parser->m_endElementHandler = NULL; + parser->m_characterDataHandler = NULL; + parser->m_processingInstructionHandler = NULL; + parser->m_commentHandler = NULL; + parser->m_startCdataSectionHandler = NULL; + parser->m_endCdataSectionHandler = NULL; + parser->m_defaultHandler = NULL; + parser->m_startDoctypeDeclHandler = NULL; + parser->m_endDoctypeDeclHandler = NULL; + parser->m_unparsedEntityDeclHandler = NULL; + parser->m_notationDeclHandler = NULL; + parser->m_startNamespaceDeclHandler = NULL; + parser->m_endNamespaceDeclHandler = NULL; + parser->m_notStandaloneHandler = NULL; + parser->m_externalEntityRefHandler = NULL; + parser->m_externalEntityRefHandlerArg = parser; + parser->m_skippedEntityHandler = NULL; + parser->m_elementDeclHandler = NULL; + parser->m_attlistDeclHandler = NULL; + parser->m_entityDeclHandler = NULL; + parser->m_xmlDeclHandler = NULL; + parser->m_bufferPtr = parser->m_buffer; + parser->m_bufferEnd = parser->m_buffer; + parser->m_parseEndByteIndex = 0; + parser->m_parseEndPtr = NULL; + parser->m_declElementType = NULL; + parser->m_declAttributeId = NULL; + parser->m_declEntity = NULL; + parser->m_doctypeName = NULL; + parser->m_doctypeSysid = NULL; + parser->m_doctypePubid = NULL; + parser->m_declAttributeType = NULL; + parser->m_declNotationName = NULL; + parser->m_declNotationPublicId = NULL; + parser->m_declAttributeIsCdata = XML_FALSE; + parser->m_declAttributeIsId = XML_FALSE; + memset(&parser->m_position, 0, sizeof(POSITION)); + parser->m_errorCode = XML_ERROR_NONE; + parser->m_eventPtr = NULL; + parser->m_eventEndPtr = NULL; + parser->m_positionPtr = NULL; + parser->m_openInternalEntities = NULL; + parser->m_defaultExpandInternalEntities = XML_TRUE; + parser->m_tagLevel = 0; + parser->m_tagStack = NULL; + parser->m_inheritedBindings = NULL; + parser->m_nSpecifiedAtts = 0; + parser->m_unknownEncodingMem = NULL; + parser->m_unknownEncodingRelease = NULL; + parser->m_unknownEncodingData = NULL; + parser->m_parentParser = NULL; + parser->m_parsingStatus.parsing = XML_INITIALIZED; #ifdef XML_DTD - isParamEntity = XML_FALSE; - useForeignDTD = XML_FALSE; - paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER; + parser->m_isParamEntity = XML_FALSE; + parser->m_useForeignDTD = XML_FALSE; + parser->m_paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER; #endif - hash_secret_salt = 0; + parser->m_hash_secret_salt = 0; } -/* moves list of bindings to freeBindingList */ +/* moves list of bindings to m_freeBindingList */ static void FASTCALL moveToFreeBindingList(XML_Parser parser, BINDING *bindings) { while (bindings) { BINDING *b = bindings; bindings = bindings->nextTagBinding; - b->nextTagBinding = freeBindingList; - freeBindingList = b; + b->nextTagBinding = parser->m_freeBindingList; + parser->m_freeBindingList = b; } } @@ -1220,36 +1130,36 @@ XML_ParserReset(XML_Parser parser, const XML_Char *encodingName) if (parser == NULL) return XML_FALSE; - if (parentParser) + if (parser->m_parentParser) return XML_FALSE; - /* move tagStack to freeTagList */ - tStk = tagStack; + /* move m_tagStack to m_freeTagList */ + tStk = parser->m_tagStack; while (tStk) { TAG *tag = tStk; tStk = tStk->parent; - tag->parent = freeTagList; + tag->parent = parser->m_freeTagList; moveToFreeBindingList(parser, tag->bindings); tag->bindings = NULL; - freeTagList = tag; + parser->m_freeTagList = tag; } - /* move openInternalEntities to freeInternalEntities */ - openEntityList = openInternalEntities; + /* move m_openInternalEntities to m_freeInternalEntities */ + openEntityList = parser->m_openInternalEntities; while (openEntityList) { OPEN_INTERNAL_ENTITY *openEntity = openEntityList; openEntityList = openEntity->next; - openEntity->next = freeInternalEntities; - freeInternalEntities = openEntity; - } - moveToFreeBindingList(parser, inheritedBindings); - FREE(unknownEncodingMem); - if (unknownEncodingRelease) - unknownEncodingRelease(unknownEncodingData); - poolClear(&tempPool); - poolClear(&temp2Pool); - FREE((void *)protocolEncodingName); - protocolEncodingName = NULL; + openEntity->next = parser->m_freeInternalEntities; + parser->m_freeInternalEntities = openEntity; + } + moveToFreeBindingList(parser, parser->m_inheritedBindings); + FREE(parser, parser->m_unknownEncodingMem); + if (parser->m_unknownEncodingRelease) + parser->m_unknownEncodingRelease(parser->m_unknownEncodingData); + poolClear(&parser->m_tempPool); + poolClear(&parser->m_temp2Pool); + FREE(parser, (void *)parser->m_protocolEncodingName); + parser->m_protocolEncodingName = NULL; parserInit(parser, encodingName); - dtdReset(_dtd, &parser->m_mem); + dtdReset(parser->m_dtd, &parser->m_mem); return XML_TRUE; } @@ -1262,19 +1172,19 @@ XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName) XXX There's no way for the caller to determine which of the XXX possible error cases caused the XML_STATUS_ERROR return. */ - if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED) + if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED) return XML_STATUS_ERROR; /* Get rid of any previous encoding name */ - FREE((void *)protocolEncodingName); + FREE(parser, (void *)parser->m_protocolEncodingName); if (encodingName == NULL) /* No new encoding name */ - protocolEncodingName = NULL; + parser->m_protocolEncodingName = NULL; else { /* Copy the new encoding name into allocated memory */ - protocolEncodingName = copyString(encodingName, &(parser->m_mem)); - if (!protocolEncodingName) + parser->m_protocolEncodingName = copyString(encodingName, &(parser->m_mem)); + if (!parser->m_protocolEncodingName) return XML_STATUS_ERROR; } return XML_STATUS_OK; @@ -1331,44 +1241,44 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser, return NULL; /* Stash the original parser contents on the stack */ - oldDtd = _dtd; - oldStartElementHandler = startElementHandler; - oldEndElementHandler = endElementHandler; - oldCharacterDataHandler = characterDataHandler; - oldProcessingInstructionHandler = processingInstructionHandler; - oldCommentHandler = commentHandler; - oldStartCdataSectionHandler = startCdataSectionHandler; - oldEndCdataSectionHandler = endCdataSectionHandler; - oldDefaultHandler = defaultHandler; - oldUnparsedEntityDeclHandler = unparsedEntityDeclHandler; - oldNotationDeclHandler = notationDeclHandler; - oldStartNamespaceDeclHandler = startNamespaceDeclHandler; - oldEndNamespaceDeclHandler = endNamespaceDeclHandler; - oldNotStandaloneHandler = notStandaloneHandler; - oldExternalEntityRefHandler = externalEntityRefHandler; - oldSkippedEntityHandler = skippedEntityHandler; - oldUnknownEncodingHandler = unknownEncodingHandler; - oldElementDeclHandler = elementDeclHandler; - oldAttlistDeclHandler = attlistDeclHandler; - oldEntityDeclHandler = entityDeclHandler; - oldXmlDeclHandler = xmlDeclHandler; - oldDeclElementType = declElementType; - - oldUserData = userData; - oldHandlerArg = handlerArg; - oldDefaultExpandInternalEntities = defaultExpandInternalEntities; - oldExternalEntityRefHandlerArg = externalEntityRefHandlerArg; + oldDtd = parser->m_dtd; + oldStartElementHandler = parser->m_startElementHandler; + oldEndElementHandler = parser->m_endElementHandler; + oldCharacterDataHandler = parser->m_characterDataHandler; + oldProcessingInstructionHandler = parser->m_processingInstructionHandler; + oldCommentHandler = parser->m_commentHandler; + oldStartCdataSectionHandler = parser->m_startCdataSectionHandler; + oldEndCdataSectionHandler = parser->m_endCdataSectionHandler; + oldDefaultHandler = parser->m_defaultHandler; + oldUnparsedEntityDeclHandler = parser->m_unparsedEntityDeclHandler; + oldNotationDeclHandler = parser->m_notationDeclHandler; + oldStartNamespaceDeclHandler = parser->m_startNamespaceDeclHandler; + oldEndNamespaceDeclHandler = parser->m_endNamespaceDeclHandler; + oldNotStandaloneHandler = parser->m_notStandaloneHandler; + oldExternalEntityRefHandler = parser->m_externalEntityRefHandler; + oldSkippedEntityHandler = parser->m_skippedEntityHandler; + oldUnknownEncodingHandler = parser->m_unknownEncodingHandler; + oldElementDeclHandler = parser->m_elementDeclHandler; + oldAttlistDeclHandler = parser->m_attlistDeclHandler; + oldEntityDeclHandler = parser->m_entityDeclHandler; + oldXmlDeclHandler = parser->m_xmlDeclHandler; + oldDeclElementType = parser->m_declElementType; + + oldUserData = parser->m_userData; + oldHandlerArg = parser->m_handlerArg; + oldDefaultExpandInternalEntities = parser->m_defaultExpandInternalEntities; + oldExternalEntityRefHandlerArg = parser->m_externalEntityRefHandlerArg; #ifdef XML_DTD - oldParamEntityParsing = paramEntityParsing; - oldInEntityValue = prologState.inEntityValue; + oldParamEntityParsing = parser->m_paramEntityParsing; + oldInEntityValue = parser->m_prologState.inEntityValue; #endif - oldns_triplets = ns_triplets; + oldns_triplets = parser->m_ns_triplets; /* Note that the new parser shares the same hash secret as the old parser, so that dtdCopy and copyEntityTable can lookup values from hash tables associated with either parser without us having to worry which hash secrets each table has. */ - oldhash_secret_salt = hash_secret_salt; + oldhash_secret_salt = parser->m_hash_secret_salt; #ifdef XML_DTD if (!context) @@ -1380,9 +1290,9 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser, here. This makes this function more painful to follow than it would be otherwise. */ - if (ns) { + if (parser->m_ns) { XML_Char tmp[2]; - *tmp = namespaceSeparator; + *tmp = parser->m_namespaceSeparator; parser = parserCreate(encodingName, &parser->m_mem, tmp, newDtd); } else { @@ -1392,62 +1302,62 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser, if (!parser) return NULL; - startElementHandler = oldStartElementHandler; - endElementHandler = oldEndElementHandler; - characterDataHandler = oldCharacterDataHandler; - processingInstructionHandler = oldProcessingInstructionHandler; - commentHandler = oldCommentHandler; - startCdataSectionHandler = oldStartCdataSectionHandler; - endCdataSectionHandler = oldEndCdataSectionHandler; - defaultHandler = oldDefaultHandler; - unparsedEntityDeclHandler = oldUnparsedEntityDeclHandler; - notationDeclHandler = oldNotationDeclHandler; - startNamespaceDeclHandler = oldStartNamespaceDeclHandler; - endNamespaceDeclHandler = oldEndNamespaceDeclHandler; - notStandaloneHandler = oldNotStandaloneHandler; - externalEntityRefHandler = oldExternalEntityRefHandler; - skippedEntityHandler = oldSkippedEntityHandler; - unknownEncodingHandler = oldUnknownEncodingHandler; - elementDeclHandler = oldElementDeclHandler; - attlistDeclHandler = oldAttlistDeclHandler; - entityDeclHandler = oldEntityDeclHandler; - xmlDeclHandler = oldXmlDeclHandler; - declElementType = oldDeclElementType; - userData = oldUserData; + parser->m_startElementHandler = oldStartElementHandler; + parser->m_endElementHandler = oldEndElementHandler; + parser->m_characterDataHandler = oldCharacterDataHandler; + parser->m_processingInstructionHandler = oldProcessingInstructionHandler; + parser->m_commentHandler = oldCommentHandler; + parser->m_startCdataSectionHandler = oldStartCdataSectionHandler; + parser->m_endCdataSectionHandler = oldEndCdataSectionHandler; + parser->m_defaultHandler = oldDefaultHandler; + parser->m_unparsedEntityDeclHandler = oldUnparsedEntityDeclHandler; + parser->m_notationDeclHandler = oldNotationDeclHandler; + parser->m_startNamespaceDeclHandler = oldStartNamespaceDeclHandler; + parser->m_endNamespaceDeclHandler = oldEndNamespaceDeclHandler; + parser->m_notStandaloneHandler = oldNotStandaloneHandler; + parser->m_externalEntityRefHandler = oldExternalEntityRefHandler; + parser->m_skippedEntityHandler = oldSkippedEntityHandler; + parser->m_unknownEncodingHandler = oldUnknownEncodingHandler; + parser->m_elementDeclHandler = oldElementDeclHandler; + parser->m_attlistDeclHandler = oldAttlistDeclHandler; + parser->m_entityDeclHandler = oldEntityDeclHandler; + parser->m_xmlDeclHandler = oldXmlDeclHandler; + parser->m_declElementType = oldDeclElementType; + parser->m_userData = oldUserData; if (oldUserData == oldHandlerArg) - handlerArg = userData; + parser->m_handlerArg = parser->m_userData; else - handlerArg = parser; + parser->m_handlerArg = parser; if (oldExternalEntityRefHandlerArg != oldParser) - externalEntityRefHandlerArg = oldExternalEntityRefHandlerArg; - defaultExpandInternalEntities = oldDefaultExpandInternalEntities; - ns_triplets = oldns_triplets; - hash_secret_salt = oldhash_secret_salt; - parentParser = oldParser; + parser->m_externalEntityRefHandlerArg = oldExternalEntityRefHandlerArg; + parser->m_defaultExpandInternalEntities = oldDefaultExpandInternalEntities; + parser->m_ns_triplets = oldns_triplets; + parser->m_hash_secret_salt = oldhash_secret_salt; + parser->m_parentParser = oldParser; #ifdef XML_DTD - paramEntityParsing = oldParamEntityParsing; - prologState.inEntityValue = oldInEntityValue; + parser->m_paramEntityParsing = oldParamEntityParsing; + parser->m_prologState.inEntityValue = oldInEntityValue; if (context) { #endif /* XML_DTD */ - if (!dtdCopy(oldParser, _dtd, oldDtd, &parser->m_mem) + if (!dtdCopy(oldParser, parser->m_dtd, oldDtd, &parser->m_mem) || !setContext(parser, context)) { XML_ParserFree(parser); return NULL; } - processor = externalEntityInitProcessor; + parser->m_processor = externalEntityInitProcessor; #ifdef XML_DTD } else { - /* The DTD instance referenced by _dtd is shared between the document's + /* The DTD instance referenced by parser->m_dtd is shared between the document's root parser and external PE parsers, therefore one does not need to call setContext. In addition, one also *must* not call setContext, because this would overwrite existing prefix->binding pointers in - _dtd with ones that get destroyed with the external PE parser. + parser->m_dtd with ones that get destroyed with the external PE parser. This would leave those prefixes with dangling pointers. */ - isParamEntity = XML_TRUE; - XmlPrologStateInitExternalEntity(&prologState); - processor = externalParEntInitProcessor; + parser->m_isParamEntity = XML_TRUE; + XmlPrologStateInitExternalEntity(&parser->m_prologState); + parser->m_processor = externalParEntInitProcessor; } #endif /* XML_DTD */ return parser; @@ -1461,8 +1371,8 @@ destroyBindings(BINDING *bindings, XML_Parser parser) if (!b) break; bindings = b->nextTagBinding; - FREE(b->uri); - FREE(b); + FREE(parser, b->uri); + FREE(parser, b); } } @@ -1473,70 +1383,70 @@ XML_ParserFree(XML_Parser parser) OPEN_INTERNAL_ENTITY *entityList; if (parser == NULL) return; - /* free tagStack and freeTagList */ - tagList = tagStack; + /* free m_tagStack and m_freeTagList */ + tagList = parser->m_tagStack; for (;;) { TAG *p; if (tagList == NULL) { - if (freeTagList == NULL) + if (parser->m_freeTagList == NULL) break; - tagList = freeTagList; - freeTagList = NULL; + tagList = parser->m_freeTagList; + parser->m_freeTagList = NULL; } p = tagList; tagList = tagList->parent; - FREE(p->buf); + FREE(parser, p->buf); destroyBindings(p->bindings, parser); - FREE(p); + FREE(parser, p); } - /* free openInternalEntities and freeInternalEntities */ - entityList = openInternalEntities; + /* free m_openInternalEntities and m_freeInternalEntities */ + entityList = parser->m_openInternalEntities; for (;;) { OPEN_INTERNAL_ENTITY *openEntity; if (entityList == NULL) { - if (freeInternalEntities == NULL) + if (parser->m_freeInternalEntities == NULL) break; - entityList = freeInternalEntities; - freeInternalEntities = NULL; + entityList = parser->m_freeInternalEntities; + parser->m_freeInternalEntities = NULL; } openEntity = entityList; entityList = entityList->next; - FREE(openEntity); + FREE(parser, openEntity); } - destroyBindings(freeBindingList, parser); - destroyBindings(inheritedBindings, parser); - poolDestroy(&tempPool); - poolDestroy(&temp2Pool); - FREE((void *)protocolEncodingName); + destroyBindings(parser->m_freeBindingList, parser); + destroyBindings(parser->m_inheritedBindings, parser); + poolDestroy(&parser->m_tempPool); + poolDestroy(&parser->m_temp2Pool); + FREE(parser, (void *)parser->m_protocolEncodingName); #ifdef XML_DTD /* external parameter entity parsers share the DTD structure parser->m_dtd with the root parser, so we must not destroy it */ - if (!isParamEntity && _dtd) + if (!parser->m_isParamEntity && parser->m_dtd) #else - if (_dtd) + if (parser->m_dtd) #endif /* XML_DTD */ - dtdDestroy(_dtd, (XML_Bool)!parentParser, &parser->m_mem); - FREE((void *)atts); + dtdDestroy(parser->m_dtd, (XML_Bool)!parser->m_parentParser, &parser->m_mem); + FREE(parser, (void *)parser->m_atts); #ifdef XML_ATTR_INFO - FREE((void *)attInfo); + FREE(parser, (void *)parser->m_attInfo); #endif - FREE(groupConnector); - FREE(buffer); - FREE(dataBuf); - FREE(nsAtts); - FREE(unknownEncodingMem); - if (unknownEncodingRelease) - unknownEncodingRelease(unknownEncodingData); - FREE(parser); + FREE(parser, parser->m_groupConnector); + FREE(parser, parser->m_buffer); + FREE(parser, parser->m_dataBuf); + FREE(parser, parser->m_nsAtts); + FREE(parser, parser->m_unknownEncodingMem); + if (parser->m_unknownEncodingRelease) + parser->m_unknownEncodingRelease(parser->m_unknownEncodingData); + FREE(parser, parser); } void XMLCALL XML_UseParserAsHandlerArg(XML_Parser parser) { if (parser != NULL) - handlerArg = parser; + parser->m_handlerArg = parser; } enum XML_Error XMLCALL @@ -1546,9 +1456,9 @@ XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD) return XML_ERROR_INVALID_ARGUMENT; #ifdef XML_DTD /* block after XML_Parse()/XML_ParseBuffer() has been called */ - if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED) + if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED) return XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING; - useForeignDTD = useDTD; + parser->m_useForeignDTD = useDTD; return XML_ERROR_NONE; #else return XML_ERROR_FEATURE_REQUIRES_XML_DTD; @@ -1561,9 +1471,9 @@ XML_SetReturnNSTriplet(XML_Parser parser, int do_nst) if (parser == NULL) return; /* block after XML_Parse()/XML_ParseBuffer() has been called */ - if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED) + if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED) return; - ns_triplets = do_nst ? XML_TRUE : XML_FALSE; + parser->m_ns_triplets = do_nst ? XML_TRUE : XML_FALSE; } void XMLCALL @@ -1571,10 +1481,10 @@ XML_SetUserData(XML_Parser parser, void *p) { if (parser == NULL) return; - if (handlerArg == userData) - handlerArg = userData = p; + if (parser->m_handlerArg == parser->m_userData) + parser->m_handlerArg = parser->m_userData = p; else - userData = p; + parser->m_userData = p; } enum XML_Status XMLCALL @@ -1583,13 +1493,13 @@ XML_SetBase(XML_Parser parser, const XML_Char *p) if (parser == NULL) return XML_STATUS_ERROR; if (p) { - p = poolCopyString(&_dtd->pool, p); + p = poolCopyString(&parser->m_dtd->pool, p); if (!p) return XML_STATUS_ERROR; - curBase = p; + parser->m_curBase = p; } else - curBase = NULL; + parser->m_curBase = NULL; return XML_STATUS_OK; } @@ -1598,7 +1508,7 @@ XML_GetBase(XML_Parser parser) { if (parser == NULL) return NULL; - return curBase; + return parser->m_curBase; } int XMLCALL @@ -1606,7 +1516,7 @@ XML_GetSpecifiedAttributeCount(XML_Parser parser) { if (parser == NULL) return -1; - return nSpecifiedAtts; + return parser->m_nSpecifiedAtts; } int XMLCALL @@ -1614,7 +1524,7 @@ XML_GetIdAttributeIndex(XML_Parser parser) { if (parser == NULL) return -1; - return idAttIndex; + return parser->m_idAttIndex; } #ifdef XML_ATTR_INFO @@ -1623,7 +1533,7 @@ XML_GetAttributeInfo(XML_Parser parser) { if (parser == NULL) return NULL; - return attInfo; + return parser->m_attInfo; } #endif @@ -1634,22 +1544,22 @@ XML_SetElementHandler(XML_Parser parser, { if (parser == NULL) return; - startElementHandler = start; - endElementHandler = end; + parser->m_startElementHandler = start; + parser->m_endElementHandler = end; } void XMLCALL XML_SetStartElementHandler(XML_Parser parser, XML_StartElementHandler start) { if (parser != NULL) - startElementHandler = start; + parser->m_startElementHandler = start; } void XMLCALL XML_SetEndElementHandler(XML_Parser parser, XML_EndElementHandler end) { if (parser != NULL) - endElementHandler = end; + parser->m_endElementHandler = end; } void XMLCALL @@ -1657,7 +1567,7 @@ XML_SetCharacterDataHandler(XML_Parser parser, XML_CharacterDataHandler handler) { if (parser != NULL) - characterDataHandler = handler; + parser->m_characterDataHandler = handler; } void XMLCALL @@ -1665,7 +1575,7 @@ XML_SetProcessingInstructionHandler(XML_Parser parser, XML_ProcessingInstructionHandler handler) { if (parser != NULL) - processingInstructionHandler = handler; + parser->m_processingInstructionHandler = handler; } void XMLCALL @@ -1673,7 +1583,7 @@ XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler) { if (parser != NULL) - commentHandler = handler; + parser->m_commentHandler = handler; } void XMLCALL @@ -1683,22 +1593,22 @@ XML_SetCdataSectionHandler(XML_Parser parser, { if (parser == NULL) return; - startCdataSectionHandler = start; - endCdataSectionHandler = end; + parser->m_startCdataSectionHandler = start; + parser->m_endCdataSectionHandler = end; } void XMLCALL XML_SetStartCdataSectionHandler(XML_Parser parser, XML_StartCdataSectionHandler start) { if (parser != NULL) - startCdataSectionHandler = start; + parser->m_startCdataSectionHandler = start; } void XMLCALL XML_SetEndCdataSectionHandler(XML_Parser parser, XML_EndCdataSectionHandler end) { if (parser != NULL) - endCdataSectionHandler = end; + parser->m_endCdataSectionHandler = end; } void XMLCALL @@ -1707,8 +1617,8 @@ XML_SetDefaultHandler(XML_Parser parser, { if (parser == NULL) return; - defaultHandler = handler; - defaultExpandInternalEntities = XML_FALSE; + parser->m_defaultHandler = handler; + parser->m_defaultExpandInternalEntities = XML_FALSE; } void XMLCALL @@ -1717,8 +1627,8 @@ XML_SetDefaultHandlerExpand(XML_Parser parser, { if (parser == NULL) return; - defaultHandler = handler; - defaultExpandInternalEntities = XML_TRUE; + parser->m_defaultHandler = handler; + parser->m_defaultExpandInternalEntities = XML_TRUE; } void XMLCALL @@ -1728,22 +1638,22 @@ XML_SetDoctypeDeclHandler(XML_Parser parser, { if (parser == NULL) return; - startDoctypeDeclHandler = start; - endDoctypeDeclHandler = end; + parser->m_startDoctypeDeclHandler = start; + parser->m_endDoctypeDeclHandler = end; } void XMLCALL XML_SetStartDoctypeDeclHandler(XML_Parser parser, XML_StartDoctypeDeclHandler start) { if (parser != NULL) - startDoctypeDeclHandler = start; + parser->m_startDoctypeDeclHandler = start; } void XMLCALL XML_SetEndDoctypeDeclHandler(XML_Parser parser, XML_EndDoctypeDeclHandler end) { if (parser != NULL) - endDoctypeDeclHandler = end; + parser->m_endDoctypeDeclHandler = end; } void XMLCALL @@ -1751,7 +1661,7 @@ XML_SetUnparsedEntityDeclHandler(XML_Parser parser, XML_UnparsedEntityDeclHandler handler) { if (parser != NULL) - unparsedEntityDeclHandler = handler; + parser->m_unparsedEntityDeclHandler = handler; } void XMLCALL @@ -1759,7 +1669,7 @@ XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler) { if (parser != NULL) - notationDeclHandler = handler; + parser->m_notationDeclHandler = handler; } void XMLCALL @@ -1769,22 +1679,22 @@ XML_SetNamespaceDeclHandler(XML_Parser parser, { if (parser == NULL) return; - startNamespaceDeclHandler = start; - endNamespaceDeclHandler = end; + parser->m_startNamespaceDeclHandler = start; + parser->m_endNamespaceDeclHandler = end; } void XMLCALL XML_SetStartNamespaceDeclHandler(XML_Parser parser, XML_StartNamespaceDeclHandler start) { if (parser != NULL) - startNamespaceDeclHandler = start; + parser->m_startNamespaceDeclHandler = start; } void XMLCALL XML_SetEndNamespaceDeclHandler(XML_Parser parser, XML_EndNamespaceDeclHandler end) { if (parser != NULL) - endNamespaceDeclHandler = end; + parser->m_endNamespaceDeclHandler = end; } void XMLCALL @@ -1792,7 +1702,7 @@ XML_SetNotStandaloneHandler(XML_Parser parser, XML_NotStandaloneHandler handler) { if (parser != NULL) - notStandaloneHandler = handler; + parser->m_notStandaloneHandler = handler; } void XMLCALL @@ -1800,7 +1710,7 @@ XML_SetExternalEntityRefHandler(XML_Parser parser, XML_ExternalEntityRefHandler handler) { if (parser != NULL) - externalEntityRefHandler = handler; + parser->m_externalEntityRefHandler = handler; } void XMLCALL @@ -1809,9 +1719,9 @@ XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg) if (parser == NULL) return; if (arg) - externalEntityRefHandlerArg = (XML_Parser)arg; + parser->m_externalEntityRefHandlerArg = (XML_Parser)arg; else - externalEntityRefHandlerArg = parser; + parser->m_externalEntityRefHandlerArg = parser; } void XMLCALL @@ -1819,7 +1729,7 @@ XML_SetSkippedEntityHandler(XML_Parser parser, XML_SkippedEntityHandler handler) { if (parser != NULL) - skippedEntityHandler = handler; + parser->m_skippedEntityHandler = handler; } void XMLCALL @@ -1829,8 +1739,8 @@ XML_SetUnknownEncodingHandler(XML_Parser parser, { if (parser == NULL) return; - unknownEncodingHandler = handler; - unknownEncodingHandlerData = data; + parser->m_unknownEncodingHandler = handler; + parser->m_unknownEncodingHandlerData = data; } void XMLCALL @@ -1838,7 +1748,7 @@ XML_SetElementDeclHandler(XML_Parser parser, XML_ElementDeclHandler eldecl) { if (parser != NULL) - elementDeclHandler = eldecl; + parser->m_elementDeclHandler = eldecl; } void XMLCALL @@ -1846,7 +1756,7 @@ XML_SetAttlistDeclHandler(XML_Parser parser, XML_AttlistDeclHandler attdecl) { if (parser != NULL) - attlistDeclHandler = attdecl; + parser->m_attlistDeclHandler = attdecl; } void XMLCALL @@ -1854,14 +1764,14 @@ XML_SetEntityDeclHandler(XML_Parser parser, XML_EntityDeclHandler handler) { if (parser != NULL) - entityDeclHandler = handler; + parser->m_entityDeclHandler = handler; } void XMLCALL XML_SetXmlDeclHandler(XML_Parser parser, XML_XmlDeclHandler handler) { if (parser != NULL) - xmlDeclHandler = handler; + parser->m_xmlDeclHandler = handler; } int XMLCALL @@ -1871,10 +1781,10 @@ XML_SetParamEntityParsing(XML_Parser parser, if (parser == NULL) return 0; /* block after XML_Parse()/XML_ParseBuffer() has been called */ - if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED) + if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED) return 0; #ifdef XML_DTD - paramEntityParsing = peParsing; + parser->m_paramEntityParsing = peParsing; return 1; #else return peParsing == XML_PARAM_ENTITY_PARSING_NEVER; @@ -1890,9 +1800,9 @@ XML_SetHashSalt(XML_Parser parser, if (parser->m_parentParser) return XML_SetHashSalt(parser->m_parentParser, hash_salt); /* block after XML_Parse()/XML_ParseBuffer() has been called */ - if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED) + if (parser->m_parsingStatus.parsing == XML_PARSING || parser->m_parsingStatus.parsing == XML_SUSPENDED) return 0; - hash_secret_salt = hash_salt; + parser->m_hash_secret_salt = hash_salt; return 1; } @@ -1904,37 +1814,38 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) parser->m_errorCode = XML_ERROR_INVALID_ARGUMENT; return XML_STATUS_ERROR; } - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: - errorCode = XML_ERROR_SUSPENDED; + parser->m_errorCode = XML_ERROR_SUSPENDED; return XML_STATUS_ERROR; case XML_FINISHED: - errorCode = XML_ERROR_FINISHED; + parser->m_errorCode = XML_ERROR_FINISHED; return XML_STATUS_ERROR; case XML_INITIALIZED: - if (parentParser == NULL && !startParsing(parser)) { - errorCode = XML_ERROR_NO_MEMORY; + if (parser->m_parentParser == NULL && !startParsing(parser)) { + parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: - ps_parsing = XML_PARSING; + parser->m_parsingStatus.parsing = XML_PARSING; } if (len == 0) { - ps_finalBuffer = (XML_Bool)isFinal; + parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal; if (!isFinal) return XML_STATUS_OK; - positionPtr = bufferPtr; - parseEndPtr = bufferEnd; + parser->m_positionPtr = parser->m_bufferPtr; + parser->m_parseEndPtr = parser->m_bufferEnd; /* If data are left over from last buffer, and we now know that these data are the final chunk of input, then we have to check them again to detect errors based on that fact. */ - errorCode = processor(parser, bufferPtr, parseEndPtr, &bufferPtr); + parser->m_errorCode = parser->m_processor(parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr); - if (errorCode == XML_ERROR_NONE) { - switch (ps_parsing) { + if (parser->m_errorCode == XML_ERROR_NONE) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: /* It is hard to be certain, but it seems that this case * cannot occur. This code is cleaning up a previous parse @@ -1948,54 +1859,54 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) * * LCOV_EXCL_START */ - XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position); - positionPtr = bufferPtr; + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position); + parser->m_positionPtr = parser->m_bufferPtr; return XML_STATUS_SUSPENDED; /* LCOV_EXCL_STOP */ case XML_INITIALIZED: case XML_PARSING: - ps_parsing = XML_FINISHED; + parser->m_parsingStatus.parsing = XML_FINISHED; /* fall through */ default: return XML_STATUS_OK; } } - eventEndPtr = eventPtr; - processor = errorProcessor; + parser->m_eventEndPtr = parser->m_eventPtr; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } #ifndef XML_CONTEXT_BYTES - else if (bufferPtr == bufferEnd) { + else if (parser->m_bufferPtr == parser->m_bufferEnd) { const char *end; int nLeftOver; enum XML_Status result; /* Detect overflow (a+b > MAX <==> b > MAX-a) */ - if (len > ((XML_Size)-1) / 2 - parseEndByteIndex) { - errorCode = XML_ERROR_NO_MEMORY; - eventPtr = eventEndPtr = NULL; - processor = errorProcessor; + if (len > ((XML_Size)-1) / 2 - parser->m_parseEndByteIndex) { + parser->m_errorCode = XML_ERROR_NO_MEMORY; + parser->m_eventPtr = parser->m_eventEndPtr = NULL; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } - parseEndByteIndex += len; - positionPtr = s; - ps_finalBuffer = (XML_Bool)isFinal; + parser->m_parseEndByteIndex += len; + parser->m_positionPtr = s; + parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal; - errorCode = processor(parser, s, parseEndPtr = s + len, &end); + parser->m_errorCode = parser->m_processor(parser, s, parser->m_parseEndPtr = s + len, &end); - if (errorCode != XML_ERROR_NONE) { - eventEndPtr = eventPtr; - processor = errorProcessor; + if (parser->m_errorCode != XML_ERROR_NONE) { + parser->m_eventEndPtr = parser->m_eventPtr; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } else { - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: result = XML_STATUS_SUSPENDED; break; case XML_INITIALIZED: case XML_PARSING: if (isFinal) { - ps_parsing = XML_FINISHED; + parser->m_parsingStatus.parsing = XML_FINISHED; return XML_STATUS_OK; } /* fall through */ @@ -2004,35 +1915,33 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) } } - XmlUpdatePosition(encoding, positionPtr, end, &position); + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, end, &parser->m_position); nLeftOver = s + len - end; if (nLeftOver) { - if (buffer == NULL || nLeftOver > bufferLim - buffer) { + if (parser->m_buffer == NULL || nLeftOver > parser->m_bufferLim - parser->m_buffer) { /* avoid _signed_ integer overflow */ char *temp = NULL; const int bytesToAllocate = (int)((unsigned)len * 2U); if (bytesToAllocate > 0) { - temp = (buffer == NULL - ? (char *)MALLOC(bytesToAllocate) - : (char *)REALLOC(buffer, bytesToAllocate)); + temp = (char *)REALLOC(parser, parser->m_buffer, bytesToAllocate); } if (temp == NULL) { - errorCode = XML_ERROR_NO_MEMORY; - eventPtr = eventEndPtr = NULL; - processor = errorProcessor; + parser->m_errorCode = XML_ERROR_NO_MEMORY; + parser->m_eventPtr = parser->m_eventEndPtr = NULL; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } - buffer = temp; - bufferLim = buffer + bytesToAllocate; + parser->m_buffer = temp; + parser->m_bufferLim = parser->m_buffer + bytesToAllocate; } - memcpy(buffer, end, nLeftOver); + memcpy(parser->m_buffer, end, nLeftOver); } - bufferPtr = buffer; - bufferEnd = buffer + nLeftOver; - positionPtr = bufferPtr; - parseEndPtr = bufferEnd; - eventPtr = bufferPtr; - eventEndPtr = bufferPtr; + parser->m_bufferPtr = parser->m_buffer; + parser->m_bufferEnd = parser->m_buffer + nLeftOver; + parser->m_positionPtr = parser->m_bufferPtr; + parser->m_parseEndPtr = parser->m_bufferEnd; + parser->m_eventPtr = parser->m_bufferPtr; + parser->m_eventEndPtr = parser->m_bufferPtr; return result; } #endif /* not defined XML_CONTEXT_BYTES */ @@ -2055,53 +1964,54 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal) if (parser == NULL) return XML_STATUS_ERROR; - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: - errorCode = XML_ERROR_SUSPENDED; + parser->m_errorCode = XML_ERROR_SUSPENDED; return XML_STATUS_ERROR; case XML_FINISHED: - errorCode = XML_ERROR_FINISHED; + parser->m_errorCode = XML_ERROR_FINISHED; return XML_STATUS_ERROR; case XML_INITIALIZED: - if (parentParser == NULL && !startParsing(parser)) { - errorCode = XML_ERROR_NO_MEMORY; + if (parser->m_parentParser == NULL && !startParsing(parser)) { + parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: - ps_parsing = XML_PARSING; + parser->m_parsingStatus.parsing = XML_PARSING; } - start = bufferPtr; - positionPtr = start; - bufferEnd += len; - parseEndPtr = bufferEnd; - parseEndByteIndex += len; - ps_finalBuffer = (XML_Bool)isFinal; + start = parser->m_bufferPtr; + parser->m_positionPtr = start; + parser->m_bufferEnd += len; + parser->m_parseEndPtr = parser->m_bufferEnd; + parser->m_parseEndByteIndex += len; + parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal; - errorCode = processor(parser, start, parseEndPtr, &bufferPtr); + parser->m_errorCode = parser->m_processor(parser, start, parser->m_parseEndPtr, &parser->m_bufferPtr); - if (errorCode != XML_ERROR_NONE) { - eventEndPtr = eventPtr; - processor = errorProcessor; + if (parser->m_errorCode != XML_ERROR_NONE) { + parser->m_eventEndPtr = parser->m_eventPtr; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } else { - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: result = XML_STATUS_SUSPENDED; break; case XML_INITIALIZED: case XML_PARSING: if (isFinal) { - ps_parsing = XML_FINISHED; + parser->m_parsingStatus.parsing = XML_FINISHED; return result; } default: ; /* should not happen */ } } - XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position); - positionPtr = bufferPtr; + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position); + parser->m_positionPtr = parser->m_bufferPtr; return result; } @@ -2111,52 +2021,59 @@ XML_GetBuffer(XML_Parser parser, int len) if (parser == NULL) return NULL; if (len < 0) { - errorCode = XML_ERROR_NO_MEMORY; + parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: - errorCode = XML_ERROR_SUSPENDED; + parser->m_errorCode = XML_ERROR_SUSPENDED; return NULL; case XML_FINISHED: - errorCode = XML_ERROR_FINISHED; + parser->m_errorCode = XML_ERROR_FINISHED; return NULL; default: ; } - if (len > bufferLim - bufferEnd) { + if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) { #ifdef XML_CONTEXT_BYTES int keep; #endif /* defined XML_CONTEXT_BYTES */ /* Do not invoke signed arithmetic overflow: */ - int neededSize = (int) ((unsigned)len + (unsigned)(bufferEnd - bufferPtr)); + int neededSize = (int) ((unsigned)len + + (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, + parser->m_bufferPtr)); if (neededSize < 0) { - errorCode = XML_ERROR_NO_MEMORY; + parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } #ifdef XML_CONTEXT_BYTES - keep = (int)(bufferPtr - buffer); + keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; neededSize += keep; #endif /* defined XML_CONTEXT_BYTES */ - if (neededSize <= bufferLim - buffer) { + if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) { #ifdef XML_CONTEXT_BYTES - if (keep < bufferPtr - buffer) { - int offset = (int)(bufferPtr - buffer) - keep; - memmove(buffer, &buffer[offset], bufferEnd - bufferPtr + keep); - bufferEnd -= offset; - bufferPtr -= offset; + if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) { + int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep; + /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */ + memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep); + parser->m_bufferEnd -= offset; + parser->m_bufferPtr -= offset; } #else - memmove(buffer, bufferPtr, bufferEnd - bufferPtr); - bufferEnd = buffer + (bufferEnd - bufferPtr); - bufferPtr = buffer; + if (parser->m_buffer && parser->m_bufferPtr) { + memmove(parser->m_buffer, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + parser->m_bufferPtr = parser->m_buffer; + } #endif /* not defined XML_CONTEXT_BYTES */ } else { char *newBuf; - int bufferSize = (int)(bufferLim - bufferPtr); + int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr); if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { @@ -2164,43 +2081,52 @@ XML_GetBuffer(XML_Parser parser, int len) bufferSize = (int) (2U * (unsigned) bufferSize); } while (bufferSize < neededSize && bufferSize > 0); if (bufferSize <= 0) { - errorCode = XML_ERROR_NO_MEMORY; + parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } - newBuf = (char *)MALLOC(bufferSize); + newBuf = (char *)MALLOC(parser, bufferSize); if (newBuf == 0) { - errorCode = XML_ERROR_NO_MEMORY; + parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } - bufferLim = newBuf + bufferSize; + parser->m_bufferLim = newBuf + bufferSize; #ifdef XML_CONTEXT_BYTES - if (bufferPtr) { - int keep = (int)(bufferPtr - buffer); + if (parser->m_bufferPtr) { + int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; - memcpy(newBuf, &bufferPtr[-keep], bufferEnd - bufferPtr + keep); - FREE(buffer); - buffer = newBuf; - bufferEnd = buffer + (bufferEnd - bufferPtr) + keep; - bufferPtr = buffer + keep; + memcpy(newBuf, &parser->m_bufferPtr[-keep], + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep); + FREE(parser, parser->m_buffer); + parser->m_buffer = newBuf; + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep; + parser->m_bufferPtr = parser->m_buffer + keep; } else { - bufferEnd = newBuf + (bufferEnd - bufferPtr); - bufferPtr = buffer = newBuf; + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; + parser->m_bufferPtr = parser->m_buffer = newBuf; } #else - if (bufferPtr) { - memcpy(newBuf, bufferPtr, bufferEnd - bufferPtr); - FREE(buffer); + if (parser->m_bufferPtr) { + memcpy(newBuf, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + FREE(parser, parser->m_buffer); + parser->m_bufferEnd = newBuf + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); } - bufferEnd = newBuf + (bufferEnd - bufferPtr); - bufferPtr = buffer = newBuf; + else { + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; + } + parser->m_bufferPtr = parser->m_buffer = newBuf; #endif /* not defined XML_CONTEXT_BYTES */ } - eventPtr = eventEndPtr = NULL; - positionPtr = NULL; + parser->m_eventPtr = parser->m_eventEndPtr = NULL; + parser->m_positionPtr = NULL; } - return bufferEnd; + return parser->m_bufferEnd; } enum XML_Status XMLCALL @@ -2208,29 +2134,29 @@ XML_StopParser(XML_Parser parser, XML_Bool resumable) { if (parser == NULL) return XML_STATUS_ERROR; - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: if (resumable) { - errorCode = XML_ERROR_SUSPENDED; + parser->m_errorCode = XML_ERROR_SUSPENDED; return XML_STATUS_ERROR; } - ps_parsing = XML_FINISHED; + parser->m_parsingStatus.parsing = XML_FINISHED; break; case XML_FINISHED: - errorCode = XML_ERROR_FINISHED; + parser->m_errorCode = XML_ERROR_FINISHED; return XML_STATUS_ERROR; default: if (resumable) { #ifdef XML_DTD - if (isParamEntity) { - errorCode = XML_ERROR_SUSPEND_PE; + if (parser->m_isParamEntity) { + parser->m_errorCode = XML_ERROR_SUSPEND_PE; return XML_STATUS_ERROR; } #endif - ps_parsing = XML_SUSPENDED; + parser->m_parsingStatus.parsing = XML_SUSPENDED; } else - ps_parsing = XML_FINISHED; + parser->m_parsingStatus.parsing = XML_FINISHED; } return XML_STATUS_OK; } @@ -2242,36 +2168,36 @@ XML_ResumeParser(XML_Parser parser) if (parser == NULL) return XML_STATUS_ERROR; - if (ps_parsing != XML_SUSPENDED) { - errorCode = XML_ERROR_NOT_SUSPENDED; + if (parser->m_parsingStatus.parsing != XML_SUSPENDED) { + parser->m_errorCode = XML_ERROR_NOT_SUSPENDED; return XML_STATUS_ERROR; } - ps_parsing = XML_PARSING; + parser->m_parsingStatus.parsing = XML_PARSING; - errorCode = processor(parser, bufferPtr, parseEndPtr, &bufferPtr); + parser->m_errorCode = parser->m_processor(parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr); - if (errorCode != XML_ERROR_NONE) { - eventEndPtr = eventPtr; - processor = errorProcessor; + if (parser->m_errorCode != XML_ERROR_NONE) { + parser->m_eventEndPtr = parser->m_eventPtr; + parser->m_processor = errorProcessor; return XML_STATUS_ERROR; } else { - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: result = XML_STATUS_SUSPENDED; break; case XML_INITIALIZED: case XML_PARSING: - if (ps_finalBuffer) { - ps_parsing = XML_FINISHED; + if (parser->m_parsingStatus.finalBuffer) { + parser->m_parsingStatus.parsing = XML_FINISHED; return result; } default: ; } } - XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position); - positionPtr = bufferPtr; + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_bufferPtr, &parser->m_position); + parser->m_positionPtr = parser->m_bufferPtr; return result; } @@ -2289,7 +2215,7 @@ XML_GetErrorCode(XML_Parser parser) { if (parser == NULL) return XML_ERROR_INVALID_ARGUMENT; - return errorCode; + return parser->m_errorCode; } XML_Index XMLCALL @@ -2297,8 +2223,8 @@ XML_GetCurrentByteIndex(XML_Parser parser) { if (parser == NULL) return -1; - if (eventPtr) - return (XML_Index)(parseEndByteIndex - (parseEndPtr - eventPtr)); + if (parser->m_eventPtr) + return (XML_Index)(parser->m_parseEndByteIndex - (parser->m_parseEndPtr - parser->m_eventPtr)); return -1; } @@ -2307,8 +2233,8 @@ XML_GetCurrentByteCount(XML_Parser parser) { if (parser == NULL) return 0; - if (eventEndPtr && eventPtr) - return (int)(eventEndPtr - eventPtr); + if (parser->m_eventEndPtr && parser->m_eventPtr) + return (int)(parser->m_eventEndPtr - parser->m_eventPtr); return 0; } @@ -2318,12 +2244,12 @@ XML_GetInputContext(XML_Parser parser, int *offset, int *size) #ifdef XML_CONTEXT_BYTES if (parser == NULL) return NULL; - if (eventPtr && buffer) { + if (parser->m_eventPtr && parser->m_buffer) { if (offset != NULL) - *offset = (int)(eventPtr - buffer); + *offset = (int)(parser->m_eventPtr - parser->m_buffer); if (size != NULL) - *size = (int)(bufferEnd - buffer); - return buffer; + *size = (int)(parser->m_bufferEnd - parser->m_buffer); + return parser->m_buffer; } #else (void)parser; @@ -2338,11 +2264,11 @@ XML_GetCurrentLineNumber(XML_Parser parser) { if (parser == NULL) return 0; - if (eventPtr && eventPtr >= positionPtr) { - XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); - positionPtr = eventPtr; + if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) { + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_eventPtr, &parser->m_position); + parser->m_positionPtr = parser->m_eventPtr; } - return position.lineNumber + 1; + return parser->m_position.lineNumber + 1; } XML_Size XMLCALL @@ -2350,18 +2276,18 @@ XML_GetCurrentColumnNumber(XML_Parser parser) { if (parser == NULL) return 0; - if (eventPtr && eventPtr >= positionPtr) { - XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); - positionPtr = eventPtr; + if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) { + XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, parser->m_eventPtr, &parser->m_position); + parser->m_positionPtr = parser->m_eventPtr; } - return position.columnNumber; + return parser->m_position.columnNumber; } void XMLCALL XML_FreeContentModel(XML_Parser parser, XML_Content *model) { if (parser != NULL) - FREE(model); + FREE(parser, model); } void * XMLCALL @@ -2369,7 +2295,7 @@ XML_MemMalloc(XML_Parser parser, size_t size) { if (parser == NULL) return NULL; - return MALLOC(size); + return MALLOC(parser, size); } void * XMLCALL @@ -2377,14 +2303,14 @@ XML_MemRealloc(XML_Parser parser, void *ptr, size_t size) { if (parser == NULL) return NULL; - return REALLOC(ptr, size); + return REALLOC(parser, ptr, size); } void XMLCALL XML_MemFree(XML_Parser parser, void *ptr) { if (parser != NULL) - FREE(ptr); + FREE(parser, ptr); } void XMLCALL @@ -2392,65 +2318,110 @@ XML_DefaultCurrent(XML_Parser parser) { if (parser == NULL) return; - if (defaultHandler) { - if (openInternalEntities) + if (parser->m_defaultHandler) { + if (parser->m_openInternalEntities) reportDefault(parser, - internalEncoding, - openInternalEntities->internalEventPtr, - openInternalEntities->internalEventEndPtr); + parser->m_internalEncoding, + parser->m_openInternalEntities->internalEventPtr, + parser->m_openInternalEntities->internalEventEndPtr); else - reportDefault(parser, encoding, eventPtr, eventEndPtr); + reportDefault(parser, parser->m_encoding, parser->m_eventPtr, parser->m_eventEndPtr); } } const XML_LChar * XMLCALL XML_ErrorString(enum XML_Error code) { - static const XML_LChar* const message[] = { - 0, - XML_L("out of memory"), - XML_L("syntax error"), - XML_L("no element found"), - XML_L("not well-formed (invalid token)"), - XML_L("unclosed token"), - XML_L("partial character"), - XML_L("mismatched tag"), - XML_L("duplicate attribute"), - XML_L("junk after document element"), - XML_L("illegal parameter entity reference"), - XML_L("undefined entity"), - XML_L("recursive entity reference"), - XML_L("asynchronous entity"), - XML_L("reference to invalid character number"), - XML_L("reference to binary entity"), - XML_L("reference to external entity in attribute"), - XML_L("XML or text declaration not at start of entity"), - XML_L("unknown encoding"), - XML_L("encoding specified in XML declaration is incorrect"), - XML_L("unclosed CDATA section"), - XML_L("error in processing external entity reference"), - XML_L("document is not standalone"), - XML_L("unexpected parser state - please send a bug report"), - XML_L("entity declared in parameter entity"), - XML_L("requested feature requires XML_DTD support in Expat"), - XML_L("cannot change setting once parsing has begun"), - XML_L("unbound prefix"), - XML_L("must not undeclare prefix"), - XML_L("incomplete markup in parameter entity"), - XML_L("XML declaration not well-formed"), - XML_L("text declaration not well-formed"), - XML_L("illegal character(s) in public id"), - XML_L("parser suspended"), - XML_L("parser not suspended"), - XML_L("parsing aborted"), - XML_L("parsing finished"), - XML_L("cannot suspend in external parameter entity"), - XML_L("reserved prefix (xml) must not be undeclared or bound to another namespace name"), - XML_L("reserved prefix (xmlns) must not be declared or undeclared"), - XML_L("prefix must not be bound to one of the reserved namespace names") - }; - if (code > 0 && code < sizeof(message)/sizeof(message[0])) - return message[code]; + switch (code) { + case XML_ERROR_NONE: + return NULL; + case XML_ERROR_NO_MEMORY: + return XML_L("out of memory"); + case XML_ERROR_SYNTAX: + return XML_L("syntax error"); + case XML_ERROR_NO_ELEMENTS: + return XML_L("no element found"); + case XML_ERROR_INVALID_TOKEN: + return XML_L("not well-formed (invalid token)"); + case XML_ERROR_UNCLOSED_TOKEN: + return XML_L("unclosed token"); + case XML_ERROR_PARTIAL_CHAR: + return XML_L("partial character"); + case XML_ERROR_TAG_MISMATCH: + return XML_L("mismatched tag"); + case XML_ERROR_DUPLICATE_ATTRIBUTE: + return XML_L("duplicate attribute"); + case XML_ERROR_JUNK_AFTER_DOC_ELEMENT: + return XML_L("junk after document element"); + case XML_ERROR_PARAM_ENTITY_REF: + return XML_L("illegal parameter entity reference"); + case XML_ERROR_UNDEFINED_ENTITY: + return XML_L("undefined entity"); + case XML_ERROR_RECURSIVE_ENTITY_REF: + return XML_L("recursive entity reference"); + case XML_ERROR_ASYNC_ENTITY: + return XML_L("asynchronous entity"); + case XML_ERROR_BAD_CHAR_REF: + return XML_L("reference to invalid character number"); + case XML_ERROR_BINARY_ENTITY_REF: + return XML_L("reference to binary entity"); + case XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF: + return XML_L("reference to external entity in attribute"); + case XML_ERROR_MISPLACED_XML_PI: + return XML_L("XML or text declaration not at start of entity"); + case XML_ERROR_UNKNOWN_ENCODING: + return XML_L("unknown encoding"); + case XML_ERROR_INCORRECT_ENCODING: + return XML_L("encoding specified in XML declaration is incorrect"); + case XML_ERROR_UNCLOSED_CDATA_SECTION: + return XML_L("unclosed CDATA section"); + case XML_ERROR_EXTERNAL_ENTITY_HANDLING: + return XML_L("error in processing external entity reference"); + case XML_ERROR_NOT_STANDALONE: + return XML_L("document is not standalone"); + case XML_ERROR_UNEXPECTED_STATE: + return XML_L("unexpected parser state - please send a bug report"); + case XML_ERROR_ENTITY_DECLARED_IN_PE: + return XML_L("entity declared in parameter entity"); + case XML_ERROR_FEATURE_REQUIRES_XML_DTD: + return XML_L("requested feature requires XML_DTD support in Expat"); + case XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING: + return XML_L("cannot change setting once parsing has begun"); + /* Added in 1.95.7. */ + case XML_ERROR_UNBOUND_PREFIX: + return XML_L("unbound prefix"); + /* Added in 1.95.8. */ + case XML_ERROR_UNDECLARING_PREFIX: + return XML_L("must not undeclare prefix"); + case XML_ERROR_INCOMPLETE_PE: + return XML_L("incomplete markup in parameter entity"); + case XML_ERROR_XML_DECL: + return XML_L("XML declaration not well-formed"); + case XML_ERROR_TEXT_DECL: + return XML_L("text declaration not well-formed"); + case XML_ERROR_PUBLICID: + return XML_L("illegal character(s) in public id"); + case XML_ERROR_SUSPENDED: + return XML_L("parser suspended"); + case XML_ERROR_NOT_SUSPENDED: + return XML_L("parser not suspended"); + case XML_ERROR_ABORTED: + return XML_L("parsing aborted"); + case XML_ERROR_FINISHED: + return XML_L("parsing finished"); + case XML_ERROR_SUSPEND_PE: + return XML_L("cannot suspend in external parameter entity"); + /* Added in 2.0.0. */ + case XML_ERROR_RESERVED_PREFIX_XML: + return XML_L("reserved prefix (xml) must not be undeclared or bound to another namespace name"); + case XML_ERROR_RESERVED_PREFIX_XMLNS: + return XML_L("reserved prefix (xmlns) must not be declared or undeclared"); + case XML_ERROR_RESERVED_NAMESPACE_URI: + return XML_L("prefix must not be bound to one of the reserved namespace names"); + /* Added in 2.2.5. */ + case XML_ERROR_INVALID_ARGUMENT: /* Constant added in 2.2.1, already */ + return XML_L("invalid argument"); + } return NULL; } @@ -2533,12 +2504,12 @@ XML_GetFeatureList(void) static XML_Bool storeRawNames(XML_Parser parser) { - TAG *tag = tagStack; + TAG *tag = parser->m_tagStack; while (tag) { int bufSize; int nameLen = sizeof(XML_Char) * (tag->name.strLen + 1); char *rawNameBuf = tag->buf + nameLen; - /* Stop if already stored. Since tagStack is a stack, we can stop + /* Stop if already stored. Since m_tagStack is a stack, we can stop at the first entry that has already been copied; everything below it in the stack is already been accounted for in a previous call to this function. @@ -2550,7 +2521,7 @@ storeRawNames(XML_Parser parser) */ bufSize = nameLen + ROUND_UP(tag->rawNameLength, sizeof(XML_Char)); if (bufSize > tag->bufEnd - tag->buf) { - char *temp = (char *)REALLOC(tag->buf, bufSize); + char *temp = (char *)REALLOC(parser, tag->buf, bufSize); if (temp == NULL) return XML_FALSE; /* if tag->name.str points to tag->buf (only when namespace @@ -2581,8 +2552,8 @@ contentProcessor(XML_Parser parser, const char *end, const char **endPtr) { - enum XML_Error result = doContent(parser, 0, encoding, start, end, - endPtr, (XML_Bool)!ps_finalBuffer); + enum XML_Error result = doContent(parser, 0, parser->m_encoding, start, end, + endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); if (result == XML_ERROR_NONE) { if (!storeRawNames(parser)) return XML_ERROR_NO_MEMORY; @@ -2599,7 +2570,7 @@ externalEntityInitProcessor(XML_Parser parser, enum XML_Error result = initializeEncoding(parser); if (result != XML_ERROR_NONE) return result; - processor = externalEntityInitProcessor2; + parser->m_processor = externalEntityInitProcessor2; return externalEntityInitProcessor2(parser, start, end, endPtr); } @@ -2610,7 +2581,7 @@ externalEntityInitProcessor2(XML_Parser parser, const char **endPtr) { const char *next = start; /* XmlContentTok doesn't always set the last arg */ - int tok = XmlContentTok(encoding, start, end, &next); + int tok = XmlContentTok(parser->m_encoding, start, end, &next); switch (tok) { case XML_TOK_BOM: /* If we are at the end of the buffer, this would cause the next stage, @@ -2618,28 +2589,28 @@ externalEntityInitProcessor2(XML_Parser parser, doContent (by detecting XML_TOK_NONE) without processing any xml text declaration - causing the error XML_ERROR_MISPLACED_XML_PI in doContent. */ - if (next == end && !ps_finalBuffer) { + if (next == end && !parser->m_parsingStatus.finalBuffer) { *endPtr = next; return XML_ERROR_NONE; } start = next; break; case XML_TOK_PARTIAL: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *endPtr = start; return XML_ERROR_NONE; } - eventPtr = start; + parser->m_eventPtr = start; return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *endPtr = start; return XML_ERROR_NONE; } - eventPtr = start; + parser->m_eventPtr = start; return XML_ERROR_PARTIAL_CHAR; } - processor = externalEntityInitProcessor3; + parser->m_processor = externalEntityInitProcessor3; return externalEntityInitProcessor3(parser, start, end, endPtr); } @@ -2651,9 +2622,9 @@ externalEntityInitProcessor3(XML_Parser parser, { int tok; const char *next = start; /* XmlContentTok doesn't always set the last arg */ - eventPtr = start; - tok = XmlContentTok(encoding, start, end, &next); - eventEndPtr = next; + parser->m_eventPtr = start; + tok = XmlContentTok(parser->m_encoding, start, end, &next); + parser->m_eventEndPtr = next; switch (tok) { case XML_TOK_XML_DECL: @@ -2662,7 +2633,7 @@ externalEntityInitProcessor3(XML_Parser parser, result = processXmlDecl(parser, 1, start, next); if (result != XML_ERROR_NONE) return result; - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: *endPtr = next; return XML_ERROR_NONE; @@ -2674,20 +2645,20 @@ externalEntityInitProcessor3(XML_Parser parser, } break; case XML_TOK_PARTIAL: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *endPtr = start; return XML_ERROR_NONE; } return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *endPtr = start; return XML_ERROR_NONE; } return XML_ERROR_PARTIAL_CHAR; } - processor = externalEntityContentProcessor; - tagLevel = 1; + parser->m_processor = externalEntityContentProcessor; + parser->m_tagLevel = 1; return externalEntityContentProcessor(parser, start, end, endPtr); } @@ -2697,8 +2668,8 @@ externalEntityContentProcessor(XML_Parser parser, const char *end, const char **endPtr) { - enum XML_Error result = doContent(parser, 1, encoding, start, end, - endPtr, (XML_Bool)!ps_finalBuffer); + enum XML_Error result = doContent(parser, 1, parser->m_encoding, start, end, + endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); if (result == XML_ERROR_NONE) { if (!storeRawNames(parser)) return XML_ERROR_NO_MEMORY; @@ -2716,17 +2687,17 @@ doContent(XML_Parser parser, XML_Bool haveMore) { /* save one level of indirection */ - DTD * const dtd = _dtd; + DTD * const dtd = parser->m_dtd; const char **eventPP; const char **eventEndPP; - if (enc == encoding) { - eventPP = &eventPtr; - eventEndPP = &eventEndPtr; + if (enc == parser->m_encoding) { + eventPP = &parser->m_eventPtr; + eventEndPP = &parser->m_eventEndPtr; } else { - eventPP = &(openInternalEntities->internalEventPtr); - eventEndPP = &(openInternalEntities->internalEventEndPtr); + eventPP = &(parser->m_openInternalEntities->internalEventPtr); + eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr); } *eventPP = s; @@ -2741,18 +2712,18 @@ doContent(XML_Parser parser, return XML_ERROR_NONE; } *eventEndPP = end; - if (characterDataHandler) { + if (parser->m_characterDataHandler) { XML_Char c = 0xA; - characterDataHandler(handlerArg, &c, 1); + parser->m_characterDataHandler(parser->m_handlerArg, &c, 1); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, end); /* We are at the end of the final buffer, should we check for XML_SUSPENDED, XML_FINISHED? */ if (startTagLevel == 0) return XML_ERROR_NO_ELEMENTS; - if (tagLevel != startTagLevel) + if (parser->m_tagLevel != startTagLevel) return XML_ERROR_ASYNC_ENTITY; *nextPtr = end; return XML_ERROR_NONE; @@ -2762,7 +2733,7 @@ doContent(XML_Parser parser, return XML_ERROR_NONE; } if (startTagLevel > 0) { - if (tagLevel != startTagLevel) + if (parser->m_tagLevel != startTagLevel) return XML_ERROR_ASYNC_ENTITY; *nextPtr = s; return XML_ERROR_NONE; @@ -2791,9 +2762,9 @@ doContent(XML_Parser parser, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (ch) { - if (characterDataHandler) - characterDataHandler(handlerArg, &ch, 1); - else if (defaultHandler) + if (parser->m_characterDataHandler) + parser->m_characterDataHandler(parser->m_handlerArg, &ch, 1); + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; } @@ -2815,9 +2786,9 @@ doContent(XML_Parser parser, return XML_ERROR_ENTITY_DECLARED_IN_PE; } else if (!entity) { - if (skippedEntityHandler) - skippedEntityHandler(handlerArg, name, 0); - else if (defaultHandler) + if (parser->m_skippedEntityHandler) + parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0); + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; } @@ -2827,10 +2798,10 @@ doContent(XML_Parser parser, return XML_ERROR_BINARY_ENTITY_REF; if (entity->textPtr) { enum XML_Error result; - if (!defaultExpandInternalEntities) { - if (skippedEntityHandler) - skippedEntityHandler(handlerArg, entity->name, 0); - else if (defaultHandler) + if (!parser->m_defaultExpandInternalEntities) { + if (parser->m_skippedEntityHandler) + parser->m_skippedEntityHandler(parser->m_handlerArg, entity->name, 0); + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; } @@ -2838,22 +2809,22 @@ doContent(XML_Parser parser, if (result != XML_ERROR_NONE) return result; } - else if (externalEntityRefHandler) { + else if (parser->m_externalEntityRefHandler) { const XML_Char *context; entity->open = XML_TRUE; context = getContext(parser); entity->open = XML_FALSE; if (!context) return XML_ERROR_NO_MEMORY; - if (!externalEntityRefHandler(externalEntityRefHandlerArg, + if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg, context, entity->base, entity->systemId, entity->publicId)) return XML_ERROR_EXTERNAL_ENTITY_HANDLING; - poolDiscard(&tempPool); + poolDiscard(&parser->m_tempPool); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; } @@ -2864,29 +2835,29 @@ doContent(XML_Parser parser, TAG *tag; enum XML_Error result; XML_Char *toPtr; - if (freeTagList) { - tag = freeTagList; - freeTagList = freeTagList->parent; + if (parser->m_freeTagList) { + tag = parser->m_freeTagList; + parser->m_freeTagList = parser->m_freeTagList->parent; } else { - tag = (TAG *)MALLOC(sizeof(TAG)); + tag = (TAG *)MALLOC(parser, sizeof(TAG)); if (!tag) return XML_ERROR_NO_MEMORY; - tag->buf = (char *)MALLOC(INIT_TAG_BUF_SIZE); + tag->buf = (char *)MALLOC(parser, INIT_TAG_BUF_SIZE); if (!tag->buf) { - FREE(tag); + FREE(parser, tag); return XML_ERROR_NO_MEMORY; } tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE; } tag->bindings = NULL; - tag->parent = tagStack; - tagStack = tag; + tag->parent = parser->m_tagStack; + parser->m_tagStack = tag; tag->name.localPart = NULL; tag->name.prefix = NULL; tag->rawName = s + enc->minBytesPerChar; tag->rawNameLength = XmlNameLength(enc, tag->rawName); - ++tagLevel; + ++parser->m_tagLevel; { const char *rawNameEnd = tag->rawName + tag->rawNameLength; const char *fromPtr = tag->rawName; @@ -2904,7 +2875,7 @@ doContent(XML_Parser parser, } bufSize = (int)(tag->bufEnd - tag->buf) << 1; { - char *temp = (char *)REALLOC(tag->buf, bufSize); + char *temp = (char *)REALLOC(parser, tag->buf, bufSize); if (temp == NULL) return XML_ERROR_NO_MEMORY; tag->buf = temp; @@ -2918,12 +2889,12 @@ doContent(XML_Parser parser, result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings)); if (result) return result; - if (startElementHandler) - startElementHandler(handlerArg, tag->name.str, - (const XML_Char **)atts); - else if (defaultHandler) + if (parser->m_startElementHandler) + parser->m_startElementHandler(parser->m_handlerArg, tag->name.str, + (const XML_Char **)parser->m_atts); + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); - poolClear(&tempPool); + poolClear(&parser->m_tempPool); break; } case XML_TOK_EMPTY_ELEMENT_NO_ATTS: @@ -2935,45 +2906,49 @@ doContent(XML_Parser parser, BINDING *bindings = NULL; XML_Bool noElmHandlers = XML_TRUE; TAG_NAME name; - name.str = poolStoreString(&tempPool, enc, rawName, + name.str = poolStoreString(&parser->m_tempPool, enc, rawName, rawName + XmlNameLength(enc, rawName)); if (!name.str) return XML_ERROR_NO_MEMORY; - poolFinish(&tempPool); + poolFinish(&parser->m_tempPool); result = storeAtts(parser, enc, s, &name, &bindings); if (result != XML_ERROR_NONE) { freeBindings(parser, bindings); return result; } - poolFinish(&tempPool); - if (startElementHandler) { - startElementHandler(handlerArg, name.str, (const XML_Char **)atts); + poolFinish(&parser->m_tempPool); + if (parser->m_startElementHandler) { + parser->m_startElementHandler(parser->m_handlerArg, name.str, (const XML_Char **)parser->m_atts); noElmHandlers = XML_FALSE; } - if (endElementHandler) { - if (startElementHandler) + if (parser->m_endElementHandler) { + if (parser->m_startElementHandler) *eventPP = *eventEndPP; - endElementHandler(handlerArg, name.str); + parser->m_endElementHandler(parser->m_handlerArg, name.str); noElmHandlers = XML_FALSE; } - if (noElmHandlers && defaultHandler) + if (noElmHandlers && parser->m_defaultHandler) reportDefault(parser, enc, s, next); - poolClear(&tempPool); + poolClear(&parser->m_tempPool); freeBindings(parser, bindings); } - if (tagLevel == 0) - return epilogProcessor(parser, next, end, nextPtr); + if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) { + if (parser->m_parsingStatus.parsing == XML_SUSPENDED) + parser->m_processor = epilogProcessor; + else + return epilogProcessor(parser, next, end, nextPtr); + } break; case XML_TOK_END_TAG: - if (tagLevel == startTagLevel) + if (parser->m_tagLevel == startTagLevel) return XML_ERROR_ASYNC_ENTITY; else { int len; const char *rawName; - TAG *tag = tagStack; - tagStack = tag->parent; - tag->parent = freeTagList; - freeTagList = tag; + TAG *tag = parser->m_tagStack; + parser->m_tagStack = tag->parent; + tag->parent = parser->m_freeTagList; + parser->m_freeTagList = tag; rawName = s + enc->minBytesPerChar*2; len = XmlNameLength(enc, rawName); if (len != tag->rawNameLength @@ -2981,13 +2956,13 @@ doContent(XML_Parser parser, *eventPP = rawName; return XML_ERROR_TAG_MISMATCH; } - --tagLevel; - if (endElementHandler) { + --parser->m_tagLevel; + if (parser->m_endElementHandler) { const XML_Char *localPart; const XML_Char *prefix; XML_Char *uri; localPart = tag->name.localPart; - if (ns && localPart) { + if (parser->m_ns && localPart) { /* localPart and prefix may have been overwritten in tag->name.str, since this points to the binding->uri buffer which gets re-used; so we have to add them again @@ -2996,26 +2971,26 @@ doContent(XML_Parser parser, /* don't need to check for space - already done in storeAtts() */ while (*localPart) *uri++ = *localPart++; prefix = (XML_Char *)tag->name.prefix; - if (ns_triplets && prefix) { - *uri++ = namespaceSeparator; + if (parser->m_ns_triplets && prefix) { + *uri++ = parser->m_namespaceSeparator; while (*prefix) *uri++ = *prefix++; } *uri = XML_T('\0'); } - endElementHandler(handlerArg, tag->name.str); + parser->m_endElementHandler(parser->m_handlerArg, tag->name.str); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); while (tag->bindings) { BINDING *b = tag->bindings; - if (endNamespaceDeclHandler) - endNamespaceDeclHandler(handlerArg, b->prefix->name); + if (parser->m_endNamespaceDeclHandler) + parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name); tag->bindings = tag->bindings->nextTagBinding; - b->nextTagBinding = freeBindingList; - freeBindingList = b; + b->nextTagBinding = parser->m_freeBindingList; + parser->m_freeBindingList = b; b->prefix->binding = b->prevPrefixBinding; } - if (tagLevel == 0) + if (parser->m_tagLevel == 0) return epilogProcessor(parser, next, end, nextPtr); } break; @@ -3024,29 +2999,29 @@ doContent(XML_Parser parser, int n = XmlCharRefNumber(enc, s); if (n < 0) return XML_ERROR_BAD_CHAR_REF; - if (characterDataHandler) { + if (parser->m_characterDataHandler) { XML_Char buf[XML_ENCODE_MAX]; - characterDataHandler(handlerArg, buf, XmlEncode(n, (ICHAR *)buf)); + parser->m_characterDataHandler(parser->m_handlerArg, buf, XmlEncode(n, (ICHAR *)buf)); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); } break; case XML_TOK_XML_DECL: return XML_ERROR_MISPLACED_XML_PI; case XML_TOK_DATA_NEWLINE: - if (characterDataHandler) { + if (parser->m_characterDataHandler) { XML_Char c = 0xA; - characterDataHandler(handlerArg, &c, 1); + parser->m_characterDataHandler(parser->m_handlerArg, &c, 1); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_CDATA_SECT_OPEN: { enum XML_Error result; - if (startCdataSectionHandler) - startCdataSectionHandler(handlerArg); + if (parser->m_startCdataSectionHandler) + parser->m_startCdataSectionHandler(parser->m_handlerArg); #if 0 /* Suppose you doing a transformation on a document that involves changing only the character data. You set up a defaultHandler @@ -3060,16 +3035,16 @@ doContent(XML_Parser parser, However, now we have a start/endCdataSectionHandler, so it seems easier to let the user deal with this. */ - else if (characterDataHandler) - characterDataHandler(handlerArg, dataBuf, 0); + else if (parser->m_characterDataHandler) + parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf, 0); #endif - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); result = doCdataSection(parser, enc, &next, end, nextPtr, haveMore); if (result != XML_ERROR_NONE) return result; else if (!next) { - processor = cdataSectionProcessor; + parser->m_processor = cdataSectionProcessor; return result; } } @@ -3079,19 +3054,19 @@ doContent(XML_Parser parser, *nextPtr = s; return XML_ERROR_NONE; } - if (characterDataHandler) { + if (parser->m_characterDataHandler) { if (MUST_CONVERT(enc, s)) { - ICHAR *dataPtr = (ICHAR *)dataBuf; - XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd); - characterDataHandler(handlerArg, dataBuf, - (int)(dataPtr - (ICHAR *)dataBuf)); + ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf; + XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd); + parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf, + (int)(dataPtr - (ICHAR *)parser->m_dataBuf)); } else - characterDataHandler(handlerArg, + parser->m_characterDataHandler(parser->m_handlerArg, (XML_Char *)s, (int)((XML_Char *)end - (XML_Char *)s)); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, end); /* We are at the end of the final buffer, should we check for XML_SUSPENDED, XML_FINISHED? @@ -3100,7 +3075,7 @@ doContent(XML_Parser parser, *eventPP = end; return XML_ERROR_NO_ELEMENTS; } - if (tagLevel != startTagLevel) { + if (parser->m_tagLevel != startTagLevel) { *eventPP = end; return XML_ERROR_ASYNC_ENTITY; } @@ -3108,26 +3083,26 @@ doContent(XML_Parser parser, return XML_ERROR_NONE; case XML_TOK_DATA_CHARS: { - XML_CharacterDataHandler charDataHandler = characterDataHandler; + XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler; if (charDataHandler) { if (MUST_CONVERT(enc, s)) { for (;;) { - ICHAR *dataPtr = (ICHAR *)dataBuf; - const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd); + ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf; + const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd); *eventEndPP = s; - charDataHandler(handlerArg, dataBuf, - (int)(dataPtr - (ICHAR *)dataBuf)); + charDataHandler(parser->m_handlerArg, parser->m_dataBuf, + (int)(dataPtr - (ICHAR *)parser->m_dataBuf)); if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) break; *eventPP = s; } } else - charDataHandler(handlerArg, + charDataHandler(parser->m_handlerArg, (XML_Char *)s, (int)((XML_Char *)next - (XML_Char *)s)); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); } break; @@ -3147,13 +3122,13 @@ doContent(XML_Parser parser, * * LCOV_EXCL_START */ - if (defaultHandler) + if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; /* LCOV_EXCL_STOP */ } *eventPP = s = next; - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: *nextPtr = next; return XML_ERROR_NONE; @@ -3166,7 +3141,7 @@ doContent(XML_Parser parser, } /* This function does not call free() on the allocated memory, merely - * moving it to the parser's freeBindingList where it can be freed or + * moving it to the parser's m_freeBindingList where it can be freed or * reused as appropriate. */ static void @@ -3175,15 +3150,15 @@ freeBindings(XML_Parser parser, BINDING *bindings) while (bindings) { BINDING *b = bindings; - /* startNamespaceDeclHandler will have been called for this + /* m_startNamespaceDeclHandler will have been called for this * binding in addBindings(), so call the end handler now. */ - if (endNamespaceDeclHandler) - endNamespaceDeclHandler(handlerArg, b->prefix->name); + if (parser->m_endNamespaceDeclHandler) + parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name); bindings = bindings->nextTagBinding; - b->nextTagBinding = freeBindingList; - freeBindingList = b; + b->nextTagBinding = parser->m_freeBindingList; + parser->m_freeBindingList = b; b->prefix->binding = b->prevPrefixBinding; } } @@ -3203,7 +3178,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc, const char *attStr, TAG_NAME *tagNamePtr, BINDING **bindingsPtr) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ ELEMENT_TYPE *elementType; int nDefaultAtts; const XML_Char **appAtts; /* the attribute list for the application */ @@ -3226,43 +3201,43 @@ storeAtts(XML_Parser parser, const ENCODING *enc, sizeof(ELEMENT_TYPE)); if (!elementType) return XML_ERROR_NO_MEMORY; - if (ns && !setElementTypePrefix(parser, elementType)) + if (parser->m_ns && !setElementTypePrefix(parser, elementType)) return XML_ERROR_NO_MEMORY; } nDefaultAtts = elementType->nDefaultAtts; /* get the attributes from the tokenizer */ - n = XmlGetAttributes(enc, attStr, attsSize, atts); - if (n + nDefaultAtts > attsSize) { - int oldAttsSize = attsSize; + n = XmlGetAttributes(enc, attStr, parser->m_attsSize, parser->m_atts); + if (n + nDefaultAtts > parser->m_attsSize) { + int oldAttsSize = parser->m_attsSize; ATTRIBUTE *temp; #ifdef XML_ATTR_INFO XML_AttrInfo *temp2; #endif - attsSize = n + nDefaultAtts + INIT_ATTS_SIZE; - temp = (ATTRIBUTE *)REALLOC((void *)atts, attsSize * sizeof(ATTRIBUTE)); + parser->m_attsSize = n + nDefaultAtts + INIT_ATTS_SIZE; + temp = (ATTRIBUTE *)REALLOC(parser, (void *)parser->m_atts, parser->m_attsSize * sizeof(ATTRIBUTE)); if (temp == NULL) { - attsSize = oldAttsSize; + parser->m_attsSize = oldAttsSize; return XML_ERROR_NO_MEMORY; } - atts = temp; + parser->m_atts = temp; #ifdef XML_ATTR_INFO - temp2 = (XML_AttrInfo *)REALLOC((void *)attInfo, attsSize * sizeof(XML_AttrInfo)); + temp2 = (XML_AttrInfo *)REALLOC(parser, (void *)parser->m_attInfo, parser->m_attsSize * sizeof(XML_AttrInfo)); if (temp2 == NULL) { - attsSize = oldAttsSize; + parser->m_attsSize = oldAttsSize; return XML_ERROR_NO_MEMORY; } - attInfo = temp2; + parser->m_attInfo = temp2; #endif if (n > oldAttsSize) - XmlGetAttributes(enc, attStr, n, atts); + XmlGetAttributes(enc, attStr, n, parser->m_atts); } - appAtts = (const XML_Char **)atts; + appAtts = (const XML_Char **)parser->m_atts; for (i = 0; i < n; i++) { - ATTRIBUTE *currAtt = &atts[i]; + ATTRIBUTE *currAtt = &parser->m_atts[i]; #ifdef XML_ATTR_INFO - XML_AttrInfo *currAttInfo = &attInfo[i]; + XML_AttrInfo *currAttInfo = &parser->m_attInfo[i]; #endif /* add the name and value to the attribute list */ ATTRIBUTE_ID *attId = getAttributeId(parser, enc, currAtt->name, @@ -3271,25 +3246,25 @@ storeAtts(XML_Parser parser, const ENCODING *enc, if (!attId) return XML_ERROR_NO_MEMORY; #ifdef XML_ATTR_INFO - currAttInfo->nameStart = parseEndByteIndex - (parseEndPtr - currAtt->name); + currAttInfo->nameStart = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->name); currAttInfo->nameEnd = currAttInfo->nameStart + XmlNameLength(enc, currAtt->name); - currAttInfo->valueStart = parseEndByteIndex - - (parseEndPtr - currAtt->valuePtr); - currAttInfo->valueEnd = parseEndByteIndex - (parseEndPtr - currAtt->valueEnd); + currAttInfo->valueStart = parser->m_parseEndByteIndex - + (parser->m_parseEndPtr - currAtt->valuePtr); + currAttInfo->valueEnd = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->valueEnd); #endif /* Detect duplicate attributes by their QNames. This does not work when namespace processing is turned on and different prefixes for the same namespace are used. For this case we have a check further down. */ if ((attId->name)[-1]) { - if (enc == encoding) - eventPtr = atts[i].name; + if (enc == parser->m_encoding) + parser->m_eventPtr = parser->m_atts[i].name; return XML_ERROR_DUPLICATE_ATTRIBUTE; } (attId->name)[-1] = 1; appAtts[attIndex++] = attId->name; - if (!atts[i].normalized) { + if (!parser->m_atts[i].normalized) { enum XML_Error result; XML_Bool isCdata = XML_TRUE; @@ -3306,20 +3281,20 @@ storeAtts(XML_Parser parser, const ENCODING *enc, /* normalize the attribute value */ result = storeAttributeValue(parser, enc, isCdata, - atts[i].valuePtr, atts[i].valueEnd, - &tempPool); + parser->m_atts[i].valuePtr, parser->m_atts[i].valueEnd, + &parser->m_tempPool); if (result) return result; - appAtts[attIndex] = poolStart(&tempPool); - poolFinish(&tempPool); + appAtts[attIndex] = poolStart(&parser->m_tempPool); + poolFinish(&parser->m_tempPool); } else { /* the value did not need normalizing */ - appAtts[attIndex] = poolStoreString(&tempPool, enc, atts[i].valuePtr, - atts[i].valueEnd); + appAtts[attIndex] = poolStoreString(&parser->m_tempPool, enc, parser->m_atts[i].valuePtr, + parser->m_atts[i].valueEnd); if (appAtts[attIndex] == 0) return XML_ERROR_NO_MEMORY; - poolFinish(&tempPool); + poolFinish(&parser->m_tempPool); } /* handle prefixed attribute names */ if (attId->prefix) { @@ -3343,16 +3318,16 @@ storeAtts(XML_Parser parser, const ENCODING *enc, } /* set-up for XML_GetSpecifiedAttributeCount and XML_GetIdAttributeIndex */ - nSpecifiedAtts = attIndex; + parser->m_nSpecifiedAtts = attIndex; if (elementType->idAtt && (elementType->idAtt->name)[-1]) { for (i = 0; i < attIndex; i += 2) if (appAtts[i] == elementType->idAtt->name) { - idAttIndex = i; + parser->m_idAttIndex = i; break; } } else - idAttIndex = -1; + parser->m_idAttIndex = -1; /* do attribute defaulting */ for (i = 0; i < nDefaultAtts; i++) { @@ -3386,33 +3361,33 @@ storeAtts(XML_Parser parser, const ENCODING *enc, i = 0; if (nPrefixes) { int j; /* hash table index */ - unsigned long version = nsAttsVersion; - int nsAttsSize = (int)1 << nsAttsPower; - unsigned char oldNsAttsPower = nsAttsPower; + unsigned long version = parser->m_nsAttsVersion; + int nsAttsSize = (int)1 << parser->m_nsAttsPower; + unsigned char oldNsAttsPower = parser->m_nsAttsPower; /* size of hash table must be at least 2 * (# of prefixed attributes) */ - if ((nPrefixes << 1) >> nsAttsPower) { /* true for nsAttsPower = 0 */ + if ((nPrefixes << 1) >> parser->m_nsAttsPower) { /* true for m_nsAttsPower = 0 */ NS_ATT *temp; /* hash table size must also be a power of 2 and >= 8 */ - while (nPrefixes >> nsAttsPower++); - if (nsAttsPower < 3) - nsAttsPower = 3; - nsAttsSize = (int)1 << nsAttsPower; - temp = (NS_ATT *)REALLOC(nsAtts, nsAttsSize * sizeof(NS_ATT)); + while (nPrefixes >> parser->m_nsAttsPower++); + if (parser->m_nsAttsPower < 3) + parser->m_nsAttsPower = 3; + nsAttsSize = (int)1 << parser->m_nsAttsPower; + temp = (NS_ATT *)REALLOC(parser, parser->m_nsAtts, nsAttsSize * sizeof(NS_ATT)); if (!temp) { - /* Restore actual size of memory in nsAtts */ - nsAttsPower = oldNsAttsPower; + /* Restore actual size of memory in m_nsAtts */ + parser->m_nsAttsPower = oldNsAttsPower; return XML_ERROR_NO_MEMORY; } - nsAtts = temp; - version = 0; /* force re-initialization of nsAtts hash table */ + parser->m_nsAtts = temp; + version = 0; /* force re-initialization of m_nsAtts hash table */ } - /* using a version flag saves us from initializing nsAtts every time */ + /* using a version flag saves us from initializing m_nsAtts every time */ if (!version) { /* initialize version flags when version wraps around */ version = INIT_ATTS_VERSION; for (j = nsAttsSize; j != 0; ) - nsAtts[--j].version = version; + parser->m_nsAtts[--j].version = version; } - nsAttsVersion = --version; + parser->m_nsAttsVersion = --version; /* expand prefixed names and check for duplicates */ for (; i < attIndex; i += 2) { @@ -3452,7 +3427,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc, for (j = 0; j < b->uriLen; j++) { const XML_Char c = b->uri[j]; - if (!poolAppendChar(&tempPool, c)) + if (!poolAppendChar(&parser->m_tempPool, c)) return XML_ERROR_NO_MEMORY; } @@ -3464,7 +3439,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc, sip24_update(&sip_state, s, keylen(s) * sizeof(XML_Char)); do { /* copies null terminator */ - if (!poolAppendChar(&tempPool, *s)) + if (!poolAppendChar(&parser->m_tempPool, *s)) return XML_ERROR_NO_MEMORY; } while (*s++); @@ -3476,40 +3451,40 @@ storeAtts(XML_Parser parser, const ENCODING *enc, unsigned char step = 0; unsigned long mask = nsAttsSize - 1; j = uriHash & mask; /* index into hash table */ - while (nsAtts[j].version == version) { + while (parser->m_nsAtts[j].version == version) { /* for speed we compare stored hash values first */ - if (uriHash == nsAtts[j].hash) { - const XML_Char *s1 = poolStart(&tempPool); - const XML_Char *s2 = nsAtts[j].uriName; + if (uriHash == parser->m_nsAtts[j].hash) { + const XML_Char *s1 = poolStart(&parser->m_tempPool); + const XML_Char *s2 = parser->m_nsAtts[j].uriName; /* s1 is null terminated, but not s2 */ for (; *s1 == *s2 && *s1 != 0; s1++, s2++); if (*s1 == 0) return XML_ERROR_DUPLICATE_ATTRIBUTE; } if (!step) - step = PROBE_STEP(uriHash, mask, nsAttsPower); + step = PROBE_STEP(uriHash, mask, parser->m_nsAttsPower); j < step ? (j += nsAttsSize - step) : (j -= step); } } - if (ns_triplets) { /* append namespace separator and prefix */ - tempPool.ptr[-1] = namespaceSeparator; + if (parser->m_ns_triplets) { /* append namespace separator and prefix */ + parser->m_tempPool.ptr[-1] = parser->m_namespaceSeparator; s = b->prefix->name; do { - if (!poolAppendChar(&tempPool, *s)) + if (!poolAppendChar(&parser->m_tempPool, *s)) return XML_ERROR_NO_MEMORY; } while (*s++); } /* store expanded name in attribute list */ - s = poolStart(&tempPool); - poolFinish(&tempPool); + s = poolStart(&parser->m_tempPool); + poolFinish(&parser->m_tempPool); appAtts[i] = s; /* fill empty slot with new version, uriName and hash value */ - nsAtts[j].version = version; - nsAtts[j].hash = uriHash; - nsAtts[j].uriName = s; + parser->m_nsAtts[j].version = version; + parser->m_nsAtts[j].hash = uriHash; + parser->m_nsAtts[j].uriName = s; if (!--nPrefixes) { i += 2; @@ -3526,7 +3501,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc, for (binding = *bindingsPtr; binding; binding = binding->nextTagBinding) binding->attId->name[-1] = 0; - if (!ns) + if (!parser->m_ns) return XML_ERROR_NONE; /* expand the element type name */ @@ -3545,7 +3520,7 @@ storeAtts(XML_Parser parser, const ENCODING *enc, else return XML_ERROR_NONE; prefixLen = 0; - if (ns_triplets && binding->prefix->name) { + if (parser->m_ns_triplets && binding->prefix->name) { for (; binding->prefix->name[prefixLen++];) ; /* prefixLen includes null terminator */ } @@ -3558,24 +3533,24 @@ storeAtts(XML_Parser parser, const ENCODING *enc, n = i + binding->uriLen + prefixLen; if (n > binding->uriAlloc) { TAG *p; - uri = (XML_Char *)MALLOC((n + EXPAND_SPARE) * sizeof(XML_Char)); + uri = (XML_Char *)MALLOC(parser, (n + EXPAND_SPARE) * sizeof(XML_Char)); if (!uri) return XML_ERROR_NO_MEMORY; binding->uriAlloc = n + EXPAND_SPARE; memcpy(uri, binding->uri, binding->uriLen * sizeof(XML_Char)); - for (p = tagStack; p; p = p->parent) + for (p = parser->m_tagStack; p; p = p->parent) if (p->name.str == binding->uri) p->name.str = uri; - FREE(binding->uri); + FREE(parser, binding->uri); binding->uri = uri; } - /* if namespaceSeparator != '\0' then uri includes it already */ + /* if m_namespaceSeparator != '\0' then uri includes it already */ uri = binding->uri + binding->uriLen; memcpy(uri, localPart, i * sizeof(XML_Char)); /* we always have a namespace separator between localPart and prefix */ if (prefixLen) { uri += i - 1; - *uri = namespaceSeparator; /* replace null terminator */ + *uri = parser->m_namespaceSeparator; /* replace null terminator */ memcpy(uri + 1, binding->prefix->name, prefixLen * sizeof(XML_Char)); } tagNamePtr->str = binding->uri; @@ -3653,48 +3628,48 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId, if (isXMLNS) return XML_ERROR_RESERVED_NAMESPACE_URI; - if (namespaceSeparator) + if (parser->m_namespaceSeparator) len++; - if (freeBindingList) { - b = freeBindingList; + if (parser->m_freeBindingList) { + b = parser->m_freeBindingList; if (len > b->uriAlloc) { - XML_Char *temp = (XML_Char *)REALLOC(b->uri, + XML_Char *temp = (XML_Char *)REALLOC(parser, b->uri, sizeof(XML_Char) * (len + EXPAND_SPARE)); if (temp == NULL) return XML_ERROR_NO_MEMORY; b->uri = temp; b->uriAlloc = len + EXPAND_SPARE; } - freeBindingList = b->nextTagBinding; + parser->m_freeBindingList = b->nextTagBinding; } else { - b = (BINDING *)MALLOC(sizeof(BINDING)); + b = (BINDING *)MALLOC(parser, sizeof(BINDING)); if (!b) return XML_ERROR_NO_MEMORY; - b->uri = (XML_Char *)MALLOC(sizeof(XML_Char) * (len + EXPAND_SPARE)); + b->uri = (XML_Char *)MALLOC(parser, sizeof(XML_Char) * (len + EXPAND_SPARE)); if (!b->uri) { - FREE(b); + FREE(parser, b); return XML_ERROR_NO_MEMORY; } b->uriAlloc = len + EXPAND_SPARE; } b->uriLen = len; memcpy(b->uri, uri, len * sizeof(XML_Char)); - if (namespaceSeparator) - b->uri[len - 1] = namespaceSeparator; + if (parser->m_namespaceSeparator) + b->uri[len - 1] = parser->m_namespaceSeparator; b->prefix = prefix; b->attId = attId; b->prevPrefixBinding = prefix->binding; /* NULL binding when default namespace undeclared */ - if (*uri == XML_T('\0') && prefix == &_dtd->defaultPrefix) + if (*uri == XML_T('\0') && prefix == &parser->m_dtd->defaultPrefix) prefix->binding = NULL; else prefix->binding = b; b->nextTagBinding = *bindingsPtr; *bindingsPtr = b; /* if attId == NULL then we are not starting a namespace scope */ - if (attId && startNamespaceDeclHandler) - startNamespaceDeclHandler(handlerArg, prefix->name, + if (attId && parser->m_startNamespaceDeclHandler) + parser->m_startNamespaceDeclHandler(parser->m_handlerArg, prefix->name, prefix->binding ? uri : 0); return XML_ERROR_NONE; } @@ -3708,17 +3683,17 @@ cdataSectionProcessor(XML_Parser parser, const char *end, const char **endPtr) { - enum XML_Error result = doCdataSection(parser, encoding, &start, end, - endPtr, (XML_Bool)!ps_finalBuffer); + enum XML_Error result = doCdataSection(parser, parser->m_encoding, &start, end, + endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); if (result != XML_ERROR_NONE) return result; if (start) { - if (parentParser) { /* we are parsing an external entity */ - processor = externalEntityContentProcessor; + if (parser->m_parentParser) { /* we are parsing an external entity */ + parser->m_processor = externalEntityContentProcessor; return externalEntityContentProcessor(parser, start, end, endPtr); } else { - processor = contentProcessor; + parser->m_processor = contentProcessor; return contentProcessor(parser, start, end, endPtr); } } @@ -3739,14 +3714,14 @@ doCdataSection(XML_Parser parser, const char *s = *startPtr; const char **eventPP; const char **eventEndPP; - if (enc == encoding) { - eventPP = &eventPtr; + if (enc == parser->m_encoding) { + eventPP = &parser->m_eventPtr; *eventPP = s; - eventEndPP = &eventEndPtr; + eventEndPP = &parser->m_eventEndPtr; } else { - eventPP = &(openInternalEntities->internalEventPtr); - eventEndPP = &(openInternalEntities->internalEventEndPtr); + eventPP = &(parser->m_openInternalEntities->internalEventPtr); + eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr); } *eventPP = s; *startPtr = NULL; @@ -3757,51 +3732,51 @@ doCdataSection(XML_Parser parser, *eventEndPP = next; switch (tok) { case XML_TOK_CDATA_SECT_CLOSE: - if (endCdataSectionHandler) - endCdataSectionHandler(handlerArg); + if (parser->m_endCdataSectionHandler) + parser->m_endCdataSectionHandler(parser->m_handlerArg); #if 0 /* see comment under XML_TOK_CDATA_SECT_OPEN */ - else if (characterDataHandler) - characterDataHandler(handlerArg, dataBuf, 0); + else if (parser->m_characterDataHandler) + parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf, 0); #endif - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); *startPtr = next; *nextPtr = next; - if (ps_parsing == XML_FINISHED) + if (parser->m_parsingStatus.parsing == XML_FINISHED) return XML_ERROR_ABORTED; else return XML_ERROR_NONE; case XML_TOK_DATA_NEWLINE: - if (characterDataHandler) { + if (parser->m_characterDataHandler) { XML_Char c = 0xA; - characterDataHandler(handlerArg, &c, 1); + parser->m_characterDataHandler(parser->m_handlerArg, &c, 1); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_DATA_CHARS: { - XML_CharacterDataHandler charDataHandler = characterDataHandler; + XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler; if (charDataHandler) { if (MUST_CONVERT(enc, s)) { for (;;) { - ICHAR *dataPtr = (ICHAR *)dataBuf; - const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd); + ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf; + const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd); *eventEndPP = next; - charDataHandler(handlerArg, dataBuf, - (int)(dataPtr - (ICHAR *)dataBuf)); + charDataHandler(parser->m_handlerArg, parser->m_dataBuf, + (int)(dataPtr - (ICHAR *)parser->m_dataBuf)); if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) break; *eventPP = s; } } else - charDataHandler(handlerArg, + charDataHandler(parser->m_handlerArg, (XML_Char *)s, (int)((XML_Char *)next - (XML_Char *)s)); } - else if (defaultHandler) + else if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); } break; @@ -3835,7 +3810,7 @@ doCdataSection(XML_Parser parser, } *eventPP = s = next; - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: *nextPtr = next; return XML_ERROR_NONE; @@ -3858,12 +3833,12 @@ ignoreSectionProcessor(XML_Parser parser, const char *end, const char **endPtr) { - enum XML_Error result = doIgnoreSection(parser, encoding, &start, end, - endPtr, (XML_Bool)!ps_finalBuffer); + enum XML_Error result = doIgnoreSection(parser, parser->m_encoding, &start, end, + endPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); if (result != XML_ERROR_NONE) return result; if (start) { - processor = prologProcessor; + parser->m_processor = prologProcessor; return prologProcessor(parser, start, end, endPtr); } return result; @@ -3885,15 +3860,15 @@ doIgnoreSection(XML_Parser parser, const char *s = *startPtr; const char **eventPP; const char **eventEndPP; - if (enc == encoding) { - eventPP = &eventPtr; + if (enc == parser->m_encoding) { + eventPP = &parser->m_eventPtr; *eventPP = s; - eventEndPP = &eventEndPtr; + eventEndPP = &parser->m_eventEndPtr; } else { /* It's not entirely clear, but it seems the following two lines * of code cannot be executed. The only occasions on which 'enc' - * is not 'parser->m_encoding' are when this function is called + * is not 'encoding' are when this function is called * from the internal entity processing, and IGNORE sections are an * error in internal entities. * @@ -3902,8 +3877,8 @@ doIgnoreSection(XML_Parser parser, * * LCOV_EXCL_START */ - eventPP = &(openInternalEntities->internalEventPtr); - eventEndPP = &(openInternalEntities->internalEventEndPtr); + eventPP = &(parser->m_openInternalEntities->internalEventPtr); + eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr); /* LCOV_EXCL_STOP */ } *eventPP = s; @@ -3912,11 +3887,11 @@ doIgnoreSection(XML_Parser parser, *eventEndPP = next; switch (tok) { case XML_TOK_IGNORE_SECT: - if (defaultHandler) + if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); *startPtr = next; *nextPtr = next; - if (ps_parsing == XML_FINISHED) + if (parser->m_parsingStatus.parsing == XML_FINISHED) return XML_ERROR_ABORTED; else return XML_ERROR_NONE; @@ -3960,27 +3935,27 @@ initializeEncoding(XML_Parser parser) #ifdef XML_UNICODE char encodingBuf[128]; /* See comments abount `protoclEncodingName` in parserInit() */ - if (!protocolEncodingName) + if (!parser->m_protocolEncodingName) s = NULL; else { int i; - for (i = 0; protocolEncodingName[i]; i++) { + for (i = 0; parser->m_protocolEncodingName[i]; i++) { if (i == sizeof(encodingBuf) - 1 - || (protocolEncodingName[i] & ~0x7f) != 0) { + || (parser->m_protocolEncodingName[i] & ~0x7f) != 0) { encodingBuf[0] = '\0'; break; } - encodingBuf[i] = (char)protocolEncodingName[i]; + encodingBuf[i] = (char)parser->m_protocolEncodingName[i]; } encodingBuf[i] = '\0'; s = encodingBuf; } #else - s = protocolEncodingName; + s = parser->m_protocolEncodingName; #endif - if ((ns ? XmlInitEncodingNS : XmlInitEncoding)(&initEncoding, &encoding, s)) + if ((parser->m_ns ? XmlInitEncodingNS : XmlInitEncoding)(&parser->m_initEncoding, &parser->m_encoding, s)) return XML_ERROR_NONE; - return handleUnknownEncoding(parser, protocolEncodingName); + return handleUnknownEncoding(parser, parser->m_protocolEncodingName); } static enum XML_Error @@ -3994,13 +3969,13 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity, const char *versionend; const XML_Char *storedversion = NULL; int standalone = -1; - if (!(ns + if (!(parser->m_ns ? XmlParseXmlDeclNS : XmlParseXmlDecl)(isGeneralTextEntity, - encoding, + parser->m_encoding, s, next, - &eventPtr, + &parser->m_eventPtr, &version, &versionend, &encodingName, @@ -4012,69 +3987,69 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity, return XML_ERROR_XML_DECL; } if (!isGeneralTextEntity && standalone == 1) { - _dtd->standalone = XML_TRUE; + parser->m_dtd->standalone = XML_TRUE; #ifdef XML_DTD - if (paramEntityParsing == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE) - paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER; + if (parser->m_paramEntityParsing == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE) + parser->m_paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER; #endif /* XML_DTD */ } - if (xmlDeclHandler) { + if (parser->m_xmlDeclHandler) { if (encodingName != NULL) { - storedEncName = poolStoreString(&temp2Pool, - encoding, + storedEncName = poolStoreString(&parser->m_temp2Pool, + parser->m_encoding, encodingName, encodingName - + XmlNameLength(encoding, encodingName)); + + XmlNameLength(parser->m_encoding, encodingName)); if (!storedEncName) return XML_ERROR_NO_MEMORY; - poolFinish(&temp2Pool); + poolFinish(&parser->m_temp2Pool); } if (version) { - storedversion = poolStoreString(&temp2Pool, - encoding, + storedversion = poolStoreString(&parser->m_temp2Pool, + parser->m_encoding, version, - versionend - encoding->minBytesPerChar); + versionend - parser->m_encoding->minBytesPerChar); if (!storedversion) return XML_ERROR_NO_MEMORY; } - xmlDeclHandler(handlerArg, storedversion, storedEncName, standalone); + parser->m_xmlDeclHandler(parser->m_handlerArg, storedversion, storedEncName, standalone); } - else if (defaultHandler) - reportDefault(parser, encoding, s, next); - if (protocolEncodingName == NULL) { + else if (parser->m_defaultHandler) + reportDefault(parser, parser->m_encoding, s, next); + if (parser->m_protocolEncodingName == NULL) { if (newEncoding) { /* Check that the specified encoding does not conflict with what * the parser has already deduced. Do we have the same number * of bytes in the smallest representation of a character? If * this is UTF-16, is it the same endianness? */ - if (newEncoding->minBytesPerChar != encoding->minBytesPerChar + if (newEncoding->minBytesPerChar != parser->m_encoding->minBytesPerChar || (newEncoding->minBytesPerChar == 2 && - newEncoding != encoding)) { - eventPtr = encodingName; + newEncoding != parser->m_encoding)) { + parser->m_eventPtr = encodingName; return XML_ERROR_INCORRECT_ENCODING; } - encoding = newEncoding; + parser->m_encoding = newEncoding; } else if (encodingName) { enum XML_Error result; if (!storedEncName) { storedEncName = poolStoreString( - &temp2Pool, encoding, encodingName, - encodingName + XmlNameLength(encoding, encodingName)); + &parser->m_temp2Pool, parser->m_encoding, encodingName, + encodingName + XmlNameLength(parser->m_encoding, encodingName)); if (!storedEncName) return XML_ERROR_NO_MEMORY; } result = handleUnknownEncoding(parser, storedEncName); - poolClear(&temp2Pool); + poolClear(&parser->m_temp2Pool); if (result == XML_ERROR_UNKNOWN_ENCODING) - eventPtr = encodingName; + parser->m_eventPtr = encodingName; return result; } } if (storedEncName || storedversion) - poolClear(&temp2Pool); + poolClear(&parser->m_temp2Pool); return XML_ERROR_NONE; } @@ -4082,7 +4057,7 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity, static enum XML_Error handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName) { - if (unknownEncodingHandler) { + if (parser->m_unknownEncodingHandler) { XML_Encoding info; int i; for (i = 0; i < 256; i++) @@ -4090,25 +4065,25 @@ handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName) info.convert = NULL; info.data = NULL; info.release = NULL; - if (unknownEncodingHandler(unknownEncodingHandlerData, encodingName, + if (parser->m_unknownEncodingHandler(parser->m_unknownEncodingHandlerData, encodingName, &info)) { ENCODING *enc; - unknownEncodingMem = MALLOC(XmlSizeOfUnknownEncoding()); - if (!unknownEncodingMem) { + parser->m_unknownEncodingMem = MALLOC(parser, XmlSizeOfUnknownEncoding()); + if (!parser->m_unknownEncodingMem) { if (info.release) info.release(info.data); return XML_ERROR_NO_MEMORY; } - enc = (ns + enc = (parser->m_ns ? XmlInitUnknownEncodingNS - : XmlInitUnknownEncoding)(unknownEncodingMem, + : XmlInitUnknownEncoding)(parser->m_unknownEncodingMem, info.map, info.convert, info.data); if (enc) { - unknownEncodingData = info.data; - unknownEncodingRelease = info.release; - encoding = enc; + parser->m_unknownEncodingData = info.data; + parser->m_unknownEncodingRelease = info.release; + parser->m_encoding = enc; return XML_ERROR_NONE; } } @@ -4127,7 +4102,7 @@ prologInitProcessor(XML_Parser parser, enum XML_Error result = initializeEncoding(parser); if (result != XML_ERROR_NONE) return result; - processor = prologProcessor; + parser->m_processor = prologProcessor; return prologProcessor(parser, s, end, nextPtr); } @@ -4145,14 +4120,14 @@ externalParEntInitProcessor(XML_Parser parser, /* we know now that XML_Parse(Buffer) has been called, so we consider the external parameter entity read */ - _dtd->paramEntityRead = XML_TRUE; + parser->m_dtd->paramEntityRead = XML_TRUE; - if (prologState.inEntityValue) { - processor = entityValueInitProcessor; + if (parser->m_prologState.inEntityValue) { + parser->m_processor = entityValueInitProcessor; return entityValueInitProcessor(parser, s, end, nextPtr); } else { - processor = externalParEntProcessor; + parser->m_processor = externalParEntProcessor; return externalParEntProcessor(parser, s, end, nextPtr); } } @@ -4166,13 +4141,13 @@ entityValueInitProcessor(XML_Parser parser, int tok; const char *start = s; const char *next = start; - eventPtr = start; + parser->m_eventPtr = start; for (;;) { - tok = XmlPrologTok(encoding, start, end, &next); - eventEndPtr = next; + tok = XmlPrologTok(parser->m_encoding, start, end, &next); + parser->m_eventEndPtr = next; if (tok <= 0) { - if (!ps_finalBuffer && tok != XML_TOK_INVALID) { + if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) { *nextPtr = s; return XML_ERROR_NONE; } @@ -4188,23 +4163,23 @@ entityValueInitProcessor(XML_Parser parser, break; } /* found end of entity value - can store it now */ - return storeEntityValue(parser, encoding, s, end); + return storeEntityValue(parser, parser->m_encoding, s, end); } else if (tok == XML_TOK_XML_DECL) { enum XML_Error result; result = processXmlDecl(parser, 0, start, next); if (result != XML_ERROR_NONE) return result; - /* At this point, ps_parsing cannot be XML_SUSPENDED. For that + /* At this point, m_parsingStatus.parsing cannot be XML_SUSPENDED. For that * to happen, a parameter entity parsing handler must have * attempted to suspend the parser, which fails and raises an * error. The parser can be aborted, but can't be suspended. */ - if (ps_parsing == XML_FINISHED) + if (parser->m_parsingStatus.parsing == XML_FINISHED) return XML_ERROR_ABORTED; *nextPtr = next; /* stop scanning for text declaration - we found one */ - processor = entityValueProcessor; + parser->m_processor = entityValueProcessor; return entityValueProcessor(parser, next, end, nextPtr); } /* If we are at the end of the buffer, this would cause XmlPrologTok to @@ -4214,7 +4189,7 @@ entityValueInitProcessor(XML_Parser parser, then, when this routine is entered the next time, XmlPrologTok will return XML_TOK_INVALID, since the BOM is still in the buffer */ - else if (tok == XML_TOK_BOM && next == end && !ps_finalBuffer) { + else if (tok == XML_TOK_BOM && next == end && !parser->m_parsingStatus.finalBuffer) { *nextPtr = next; return XML_ERROR_NONE; } @@ -4227,7 +4202,7 @@ entityValueInitProcessor(XML_Parser parser, return XML_ERROR_SYNTAX; } start = next; - eventPtr = start; + parser->m_eventPtr = start; } } @@ -4240,9 +4215,9 @@ externalParEntProcessor(XML_Parser parser, const char *next = s; int tok; - tok = XmlPrologTok(encoding, s, end, &next); + tok = XmlPrologTok(parser->m_encoding, s, end, &next); if (tok <= 0) { - if (!ps_finalBuffer && tok != XML_TOK_INVALID) { + if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) { *nextPtr = s; return XML_ERROR_NONE; } @@ -4264,12 +4239,12 @@ externalParEntProcessor(XML_Parser parser, */ else if (tok == XML_TOK_BOM) { s = next; - tok = XmlPrologTok(encoding, s, end, &next); + tok = XmlPrologTok(parser->m_encoding, s, end, &next); } - processor = prologProcessor; - return doProlog(parser, encoding, s, end, tok, next, - nextPtr, (XML_Bool)!ps_finalBuffer); + parser->m_processor = prologProcessor; + return doProlog(parser, parser->m_encoding, s, end, tok, next, + nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); } static enum XML_Error PTRCALL @@ -4280,13 +4255,13 @@ entityValueProcessor(XML_Parser parser, { const char *start = s; const char *next = s; - const ENCODING *enc = encoding; + const ENCODING *enc = parser->m_encoding; int tok; for (;;) { tok = XmlPrologTok(enc, start, end, &next); if (tok <= 0) { - if (!ps_finalBuffer && tok != XML_TOK_INVALID) { + if (!parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) { *nextPtr = s; return XML_ERROR_NONE; } @@ -4317,9 +4292,9 @@ prologProcessor(XML_Parser parser, const char **nextPtr) { const char *next = s; - int tok = XmlPrologTok(encoding, s, end, &next); - return doProlog(parser, encoding, s, end, tok, next, - nextPtr, (XML_Bool)!ps_finalBuffer); + int tok = XmlPrologTok(parser->m_encoding, s, end, &next); + return doProlog(parser, parser->m_encoding, s, end, tok, next, + nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); } static enum XML_Error @@ -4356,19 +4331,19 @@ doProlog(XML_Parser parser, static const XML_Char enumValueStart[] = { ASCII_LPAREN, '\0' }; /* save one level of indirection */ - DTD * const dtd = _dtd; + DTD * const dtd = parser->m_dtd; const char **eventPP; const char **eventEndPP; enum XML_Content_Quant quant; - if (enc == encoding) { - eventPP = &eventPtr; - eventEndPP = &eventEndPtr; + if (enc == parser->m_encoding) { + eventPP = &parser->m_eventPtr; + eventEndPP = &parser->m_eventEndPtr; } else { - eventPP = &(openInternalEntities->internalEventPtr); - eventEndPP = &(openInternalEntities->internalEventEndPtr); + eventPP = &(parser->m_openInternalEntities->internalEventPtr); + eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr); } for (;;) { @@ -4395,7 +4370,7 @@ doProlog(XML_Parser parser, case XML_TOK_NONE: #ifdef XML_DTD /* for internal PE NOT referenced between declarations */ - if (enc != encoding && !openInternalEntities->betweenDecl) { + if (enc != parser->m_encoding && !parser->m_openInternalEntities->betweenDecl) { *nextPtr = s; return XML_ERROR_NONE; } @@ -4403,8 +4378,8 @@ doProlog(XML_Parser parser, complete markup, not only for external PEs, but also for internal PEs if the reference occurs between declarations. */ - if (isParamEntity || enc != encoding) { - if (XmlTokenRole(&prologState, XML_TOK_NONE, end, end, enc) + if (parser->m_isParamEntity || enc != parser->m_encoding) { + if (XmlTokenRole(&parser->m_prologState, XML_TOK_NONE, end, end, enc) == XML_ROLE_ERROR) return XML_ERROR_INCOMPLETE_PE; *nextPtr = s; @@ -4418,34 +4393,34 @@ doProlog(XML_Parser parser, break; } } - role = XmlTokenRole(&prologState, tok, s, next, enc); + role = XmlTokenRole(&parser->m_prologState, tok, s, next, enc); switch (role) { case XML_ROLE_XML_DECL: { enum XML_Error result = processXmlDecl(parser, 0, s, next); if (result != XML_ERROR_NONE) return result; - enc = encoding; + enc = parser->m_encoding; handleDefault = XML_FALSE; } break; case XML_ROLE_DOCTYPE_NAME: - if (startDoctypeDeclHandler) { - doctypeName = poolStoreString(&tempPool, enc, s, next); - if (!doctypeName) + if (parser->m_startDoctypeDeclHandler) { + parser->m_doctypeName = poolStoreString(&parser->m_tempPool, enc, s, next); + if (!parser->m_doctypeName) return XML_ERROR_NO_MEMORY; - poolFinish(&tempPool); - doctypePubid = NULL; + poolFinish(&parser->m_tempPool); + parser->m_doctypePubid = NULL; handleDefault = XML_FALSE; } - doctypeSysid = NULL; /* always initialize to NULL */ + parser->m_doctypeSysid = NULL; /* always initialize to NULL */ break; case XML_ROLE_DOCTYPE_INTERNAL_SUBSET: - if (startDoctypeDeclHandler) { - startDoctypeDeclHandler(handlerArg, doctypeName, doctypeSysid, - doctypePubid, 1); - doctypeName = NULL; - poolClear(&tempPool); + if (parser->m_startDoctypeDeclHandler) { + parser->m_startDoctypeDeclHandler(parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid, + parser->m_doctypePubid, 1); + parser->m_doctypeName = NULL; + poolClear(&parser->m_tempPool); handleDefault = XML_FALSE; } break; @@ -4455,34 +4430,34 @@ doProlog(XML_Parser parser, enum XML_Error result = processXmlDecl(parser, 1, s, next); if (result != XML_ERROR_NONE) return result; - enc = encoding; + enc = parser->m_encoding; handleDefault = XML_FALSE; } break; #endif /* XML_DTD */ case XML_ROLE_DOCTYPE_PUBLIC_ID: #ifdef XML_DTD - useForeignDTD = XML_FALSE; - declEntity = (ENTITY *)lookup(parser, + parser->m_useForeignDTD = XML_FALSE; + parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY)); - if (!declEntity) + if (!parser->m_declEntity) return XML_ERROR_NO_MEMORY; #endif /* XML_DTD */ dtd->hasParamEntityRefs = XML_TRUE; - if (startDoctypeDeclHandler) { + if (parser->m_startDoctypeDeclHandler) { XML_Char *pubId; if (!XmlIsPublicId(enc, s, next, eventPP)) return XML_ERROR_PUBLICID; - pubId = poolStoreString(&tempPool, enc, + pubId = poolStoreString(&parser->m_tempPool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!pubId) return XML_ERROR_NO_MEMORY; normalizePublicId(pubId); - poolFinish(&tempPool); - doctypePubid = pubId; + poolFinish(&parser->m_tempPool); + parser->m_doctypePubid = pubId; handleDefault = XML_FALSE; goto alreadyChecked; } @@ -4491,7 +4466,7 @@ doProlog(XML_Parser parser, if (!XmlIsPublicId(enc, s, next, eventPP)) return XML_ERROR_PUBLICID; alreadyChecked: - if (dtd->keepProcessing && declEntity) { + if (dtd->keepProcessing && parser->m_declEntity) { XML_Char *tem = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar, @@ -4499,28 +4474,31 @@ doProlog(XML_Parser parser, if (!tem) return XML_ERROR_NO_MEMORY; normalizePublicId(tem); - declEntity->publicId = tem; + parser->m_declEntity->publicId = tem; poolFinish(&dtd->pool); - if (entityDeclHandler) + /* Don't suppress the default handler if we fell through from + * the XML_ROLE_DOCTYPE_PUBLIC_ID case. + */ + if (parser->m_entityDeclHandler && role == XML_ROLE_ENTITY_PUBLIC_ID) handleDefault = XML_FALSE; } break; case XML_ROLE_DOCTYPE_CLOSE: - if (doctypeName) { - startDoctypeDeclHandler(handlerArg, doctypeName, - doctypeSysid, doctypePubid, 0); - poolClear(&tempPool); + if (parser->m_doctypeName) { + parser->m_startDoctypeDeclHandler(parser->m_handlerArg, parser->m_doctypeName, + parser->m_doctypeSysid, parser->m_doctypePubid, 0); + poolClear(&parser->m_tempPool); handleDefault = XML_FALSE; } - /* doctypeSysid will be non-NULL in the case of a previous - XML_ROLE_DOCTYPE_SYSTEM_ID, even if startDoctypeDeclHandler + /* parser->m_doctypeSysid will be non-NULL in the case of a previous + XML_ROLE_DOCTYPE_SYSTEM_ID, even if parser->m_startDoctypeDeclHandler was not set, indicating an external subset */ #ifdef XML_DTD - if (doctypeSysid || useForeignDTD) { + if (parser->m_doctypeSysid || parser->m_useForeignDTD) { XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs; dtd->hasParamEntityRefs = XML_TRUE; - if (paramEntityParsing && externalEntityRefHandler) { + if (parser->m_paramEntityParsing && parser->m_externalEntityRefHandler) { ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities, externalSubsetName, @@ -4533,10 +4511,10 @@ doProlog(XML_Parser parser, */ return XML_ERROR_NO_MEMORY; /* LCOV_EXCL_LINE */ } - if (useForeignDTD) - entity->base = curBase; + if (parser->m_useForeignDTD) + entity->base = parser->m_curBase; dtd->paramEntityRead = XML_FALSE; - if (!externalEntityRefHandler(externalEntityRefHandlerArg, + if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg, 0, entity->base, entity->systemId, @@ -4544,22 +4522,22 @@ doProlog(XML_Parser parser, return XML_ERROR_EXTERNAL_ENTITY_HANDLING; if (dtd->paramEntityRead) { if (!dtd->standalone && - notStandaloneHandler && - !notStandaloneHandler(handlerArg)) + parser->m_notStandaloneHandler && + !parser->m_notStandaloneHandler(parser->m_handlerArg)) return XML_ERROR_NOT_STANDALONE; } /* if we didn't read the foreign DTD then this means that there is no external subset and we must reset dtd->hasParamEntityRefs */ - else if (!doctypeSysid) + else if (!parser->m_doctypeSysid) dtd->hasParamEntityRefs = hadParamEntityRefs; /* end of DTD - no need to update dtd->keepProcessing */ } - useForeignDTD = XML_FALSE; + parser->m_useForeignDTD = XML_FALSE; } #endif /* XML_DTD */ - if (endDoctypeDeclHandler) { - endDoctypeDeclHandler(handlerArg); + if (parser->m_endDoctypeDeclHandler) { + parser->m_endDoctypeDeclHandler(parser->m_handlerArg); handleDefault = XML_FALSE; } break; @@ -4568,18 +4546,18 @@ doProlog(XML_Parser parser, /* if there is no DOCTYPE declaration then now is the last chance to read the foreign DTD */ - if (useForeignDTD) { + if (parser->m_useForeignDTD) { XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs; dtd->hasParamEntityRefs = XML_TRUE; - if (paramEntityParsing && externalEntityRefHandler) { + if (parser->m_paramEntityParsing && parser->m_externalEntityRefHandler) { ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY)); if (!entity) return XML_ERROR_NO_MEMORY; - entity->base = curBase; + entity->base = parser->m_curBase; dtd->paramEntityRead = XML_FALSE; - if (!externalEntityRefHandler(externalEntityRefHandlerArg, + if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg, 0, entity->base, entity->systemId, @@ -4587,8 +4565,8 @@ doProlog(XML_Parser parser, return XML_ERROR_EXTERNAL_ENTITY_HANDLING; if (dtd->paramEntityRead) { if (!dtd->standalone && - notStandaloneHandler && - !notStandaloneHandler(handlerArg)) + parser->m_notStandaloneHandler && + !parser->m_notStandaloneHandler(parser->m_handlerArg)) return XML_ERROR_NOT_STANDALONE; } /* if we didn't read the foreign DTD then this means that there @@ -4600,55 +4578,55 @@ doProlog(XML_Parser parser, } } #endif /* XML_DTD */ - processor = contentProcessor; + parser->m_processor = contentProcessor; return contentProcessor(parser, s, end, nextPtr); case XML_ROLE_ATTLIST_ELEMENT_NAME: - declElementType = getElementType(parser, enc, s, next); - if (!declElementType) + parser->m_declElementType = getElementType(parser, enc, s, next); + if (!parser->m_declElementType) return XML_ERROR_NO_MEMORY; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_NAME: - declAttributeId = getAttributeId(parser, enc, s, next); - if (!declAttributeId) + parser->m_declAttributeId = getAttributeId(parser, enc, s, next); + if (!parser->m_declAttributeId) return XML_ERROR_NO_MEMORY; - declAttributeIsCdata = XML_FALSE; - declAttributeType = NULL; - declAttributeIsId = XML_FALSE; + parser->m_declAttributeIsCdata = XML_FALSE; + parser->m_declAttributeType = NULL; + parser->m_declAttributeIsId = XML_FALSE; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_CDATA: - declAttributeIsCdata = XML_TRUE; - declAttributeType = atypeCDATA; + parser->m_declAttributeIsCdata = XML_TRUE; + parser->m_declAttributeType = atypeCDATA; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_ID: - declAttributeIsId = XML_TRUE; - declAttributeType = atypeID; + parser->m_declAttributeIsId = XML_TRUE; + parser->m_declAttributeType = atypeID; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_IDREF: - declAttributeType = atypeIDREF; + parser->m_declAttributeType = atypeIDREF; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_IDREFS: - declAttributeType = atypeIDREFS; + parser->m_declAttributeType = atypeIDREFS; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_ENTITY: - declAttributeType = atypeENTITY; + parser->m_declAttributeType = atypeENTITY; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_ENTITIES: - declAttributeType = atypeENTITIES; + parser->m_declAttributeType = atypeENTITIES; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_NMTOKEN: - declAttributeType = atypeNMTOKEN; + parser->m_declAttributeType = atypeNMTOKEN; goto checkAttListDeclHandler; case XML_ROLE_ATTRIBUTE_TYPE_NMTOKENS: - declAttributeType = atypeNMTOKENS; + parser->m_declAttributeType = atypeNMTOKENS; checkAttListDeclHandler: - if (dtd->keepProcessing && attlistDeclHandler) + if (dtd->keepProcessing && parser->m_attlistDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_ATTRIBUTE_ENUM_VALUE: case XML_ROLE_ATTRIBUTE_NOTATION_VALUE: - if (dtd->keepProcessing && attlistDeclHandler) { + if (dtd->keepProcessing && parser->m_attlistDeclHandler) { const XML_Char *prefix; - if (declAttributeType) { + if (parser->m_declAttributeType) { prefix = enumValueSep; } else { @@ -4656,37 +4634,37 @@ doProlog(XML_Parser parser, ? notationPrefix : enumValueStart); } - if (!poolAppendString(&tempPool, prefix)) + if (!poolAppendString(&parser->m_tempPool, prefix)) return XML_ERROR_NO_MEMORY; - if (!poolAppend(&tempPool, enc, s, next)) + if (!poolAppend(&parser->m_tempPool, enc, s, next)) return XML_ERROR_NO_MEMORY; - declAttributeType = tempPool.start; + parser->m_declAttributeType = parser->m_tempPool.start; handleDefault = XML_FALSE; } break; case XML_ROLE_IMPLIED_ATTRIBUTE_VALUE: case XML_ROLE_REQUIRED_ATTRIBUTE_VALUE: if (dtd->keepProcessing) { - if (!defineAttribute(declElementType, declAttributeId, - declAttributeIsCdata, declAttributeIsId, + if (!defineAttribute(parser->m_declElementType, parser->m_declAttributeId, + parser->m_declAttributeIsCdata, parser->m_declAttributeIsId, 0, parser)) return XML_ERROR_NO_MEMORY; - if (attlistDeclHandler && declAttributeType) { - if (*declAttributeType == XML_T(ASCII_LPAREN) - || (*declAttributeType == XML_T(ASCII_N) - && declAttributeType[1] == XML_T(ASCII_O))) { + if (parser->m_attlistDeclHandler && parser->m_declAttributeType) { + if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN) + || (*parser->m_declAttributeType == XML_T(ASCII_N) + && parser->m_declAttributeType[1] == XML_T(ASCII_O))) { /* Enumerated or Notation type */ - if (!poolAppendChar(&tempPool, XML_T(ASCII_RPAREN)) - || !poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN)) + || !poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return XML_ERROR_NO_MEMORY; - declAttributeType = tempPool.start; - poolFinish(&tempPool); + parser->m_declAttributeType = parser->m_tempPool.start; + poolFinish(&parser->m_tempPool); } *eventEndPP = s; - attlistDeclHandler(handlerArg, declElementType->name, - declAttributeId->name, declAttributeType, + parser->m_attlistDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, + parser->m_declAttributeId->name, parser->m_declAttributeType, 0, role == XML_ROLE_REQUIRED_ATTRIBUTE_VALUE); - poolClear(&tempPool); + poolClear(&parser->m_tempPool); handleDefault = XML_FALSE; } } @@ -4696,7 +4674,7 @@ doProlog(XML_Parser parser, if (dtd->keepProcessing) { const XML_Char *attVal; enum XML_Error result = - storeAttributeValue(parser, enc, declAttributeIsCdata, + storeAttributeValue(parser, enc, parser->m_declAttributeIsCdata, s + enc->minBytesPerChar, next - enc->minBytesPerChar, &dtd->pool); @@ -4705,26 +4683,26 @@ doProlog(XML_Parser parser, attVal = poolStart(&dtd->pool); poolFinish(&dtd->pool); /* ID attributes aren't allowed to have a default */ - if (!defineAttribute(declElementType, declAttributeId, - declAttributeIsCdata, XML_FALSE, attVal, parser)) + if (!defineAttribute(parser->m_declElementType, parser->m_declAttributeId, + parser->m_declAttributeIsCdata, XML_FALSE, attVal, parser)) return XML_ERROR_NO_MEMORY; - if (attlistDeclHandler && declAttributeType) { - if (*declAttributeType == XML_T(ASCII_LPAREN) - || (*declAttributeType == XML_T(ASCII_N) - && declAttributeType[1] == XML_T(ASCII_O))) { + if (parser->m_attlistDeclHandler && parser->m_declAttributeType) { + if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN) + || (*parser->m_declAttributeType == XML_T(ASCII_N) + && parser->m_declAttributeType[1] == XML_T(ASCII_O))) { /* Enumerated or Notation type */ - if (!poolAppendChar(&tempPool, XML_T(ASCII_RPAREN)) - || !poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN)) + || !poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return XML_ERROR_NO_MEMORY; - declAttributeType = tempPool.start; - poolFinish(&tempPool); + parser->m_declAttributeType = parser->m_tempPool.start; + poolFinish(&parser->m_tempPool); } *eventEndPP = s; - attlistDeclHandler(handlerArg, declElementType->name, - declAttributeId->name, declAttributeType, + parser->m_attlistDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, + parser->m_declAttributeId->name, parser->m_declAttributeType, attVal, role == XML_ROLE_FIXED_ATTRIBUTE_VALUE); - poolClear(&tempPool); + poolClear(&parser->m_tempPool); handleDefault = XML_FALSE; } } @@ -4734,18 +4712,18 @@ doProlog(XML_Parser parser, enum XML_Error result = storeEntityValue(parser, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); - if (declEntity) { - declEntity->textPtr = poolStart(&dtd->entityValuePool); - declEntity->textLen = (int)(poolLength(&dtd->entityValuePool)); + if (parser->m_declEntity) { + parser->m_declEntity->textPtr = poolStart(&dtd->entityValuePool); + parser->m_declEntity->textLen = (int)(poolLength(&dtd->entityValuePool)); poolFinish(&dtd->entityValuePool); - if (entityDeclHandler) { + if (parser->m_entityDeclHandler) { *eventEndPP = s; - entityDeclHandler(handlerArg, - declEntity->name, - declEntity->is_param, - declEntity->textPtr, - declEntity->textLen, - curBase, 0, 0, 0); + parser->m_entityDeclHandler(parser->m_handlerArg, + parser->m_declEntity->name, + parser->m_declEntity->is_param, + parser->m_declEntity->textPtr, + parser->m_declEntity->textLen, + parser->m_curBase, 0, 0, 0); handleDefault = XML_FALSE; } } @@ -4757,97 +4735,100 @@ doProlog(XML_Parser parser, break; case XML_ROLE_DOCTYPE_SYSTEM_ID: #ifdef XML_DTD - useForeignDTD = XML_FALSE; + parser->m_useForeignDTD = XML_FALSE; #endif /* XML_DTD */ dtd->hasParamEntityRefs = XML_TRUE; - if (startDoctypeDeclHandler) { - doctypeSysid = poolStoreString(&tempPool, enc, + if (parser->m_startDoctypeDeclHandler) { + parser->m_doctypeSysid = poolStoreString(&parser->m_tempPool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); - if (doctypeSysid == NULL) + if (parser->m_doctypeSysid == NULL) return XML_ERROR_NO_MEMORY; - poolFinish(&tempPool); + poolFinish(&parser->m_tempPool); handleDefault = XML_FALSE; } #ifdef XML_DTD else - /* use externalSubsetName to make doctypeSysid non-NULL - for the case where no startDoctypeDeclHandler is set */ - doctypeSysid = externalSubsetName; + /* use externalSubsetName to make parser->m_doctypeSysid non-NULL + for the case where no parser->m_startDoctypeDeclHandler is set */ + parser->m_doctypeSysid = externalSubsetName; #endif /* XML_DTD */ if (!dtd->standalone #ifdef XML_DTD - && !paramEntityParsing + && !parser->m_paramEntityParsing #endif /* XML_DTD */ - && notStandaloneHandler - && !notStandaloneHandler(handlerArg)) + && parser->m_notStandaloneHandler + && !parser->m_notStandaloneHandler(parser->m_handlerArg)) return XML_ERROR_NOT_STANDALONE; #ifndef XML_DTD break; #else /* XML_DTD */ - if (!declEntity) { - declEntity = (ENTITY *)lookup(parser, + if (!parser->m_declEntity) { + parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY)); - if (!declEntity) + if (!parser->m_declEntity) return XML_ERROR_NO_MEMORY; - declEntity->publicId = NULL; + parser->m_declEntity->publicId = NULL; } - /* fall through */ #endif /* XML_DTD */ + /* fall through */ case XML_ROLE_ENTITY_SYSTEM_ID: - if (dtd->keepProcessing && declEntity) { - declEntity->systemId = poolStoreString(&dtd->pool, enc, + if (dtd->keepProcessing && parser->m_declEntity) { + parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); - if (!declEntity->systemId) + if (!parser->m_declEntity->systemId) return XML_ERROR_NO_MEMORY; - declEntity->base = curBase; + parser->m_declEntity->base = parser->m_curBase; poolFinish(&dtd->pool); - if (entityDeclHandler) + /* Don't suppress the default handler if we fell through from + * the XML_ROLE_DOCTYPE_SYSTEM_ID case. + */ + if (parser->m_entityDeclHandler && role == XML_ROLE_ENTITY_SYSTEM_ID) handleDefault = XML_FALSE; } break; case XML_ROLE_ENTITY_COMPLETE: - if (dtd->keepProcessing && declEntity && entityDeclHandler) { + if (dtd->keepProcessing && parser->m_declEntity && parser->m_entityDeclHandler) { *eventEndPP = s; - entityDeclHandler(handlerArg, - declEntity->name, - declEntity->is_param, + parser->m_entityDeclHandler(parser->m_handlerArg, + parser->m_declEntity->name, + parser->m_declEntity->is_param, 0,0, - declEntity->base, - declEntity->systemId, - declEntity->publicId, + parser->m_declEntity->base, + parser->m_declEntity->systemId, + parser->m_declEntity->publicId, 0); handleDefault = XML_FALSE; } break; case XML_ROLE_ENTITY_NOTATION_NAME: - if (dtd->keepProcessing && declEntity) { - declEntity->notation = poolStoreString(&dtd->pool, enc, s, next); - if (!declEntity->notation) + if (dtd->keepProcessing && parser->m_declEntity) { + parser->m_declEntity->notation = poolStoreString(&dtd->pool, enc, s, next); + if (!parser->m_declEntity->notation) return XML_ERROR_NO_MEMORY; poolFinish(&dtd->pool); - if (unparsedEntityDeclHandler) { + if (parser->m_unparsedEntityDeclHandler) { *eventEndPP = s; - unparsedEntityDeclHandler(handlerArg, - declEntity->name, - declEntity->base, - declEntity->systemId, - declEntity->publicId, - declEntity->notation); + parser->m_unparsedEntityDeclHandler(parser->m_handlerArg, + parser->m_declEntity->name, + parser->m_declEntity->base, + parser->m_declEntity->systemId, + parser->m_declEntity->publicId, + parser->m_declEntity->notation); handleDefault = XML_FALSE; } - else if (entityDeclHandler) { + else if (parser->m_entityDeclHandler) { *eventEndPP = s; - entityDeclHandler(handlerArg, - declEntity->name, + parser->m_entityDeclHandler(parser->m_handlerArg, + parser->m_declEntity->name, 0,0,0, - declEntity->base, - declEntity->systemId, - declEntity->publicId, - declEntity->notation); + parser->m_declEntity->base, + parser->m_declEntity->systemId, + parser->m_declEntity->publicId, + parser->m_declEntity->notation); handleDefault = XML_FALSE; } } @@ -4855,36 +4836,36 @@ doProlog(XML_Parser parser, case XML_ROLE_GENERAL_ENTITY_NAME: { if (XmlPredefinedEntityName(enc, s, next)) { - declEntity = NULL; + parser->m_declEntity = NULL; break; } if (dtd->keepProcessing) { const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next); if (!name) return XML_ERROR_NO_MEMORY; - declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, + parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, sizeof(ENTITY)); - if (!declEntity) + if (!parser->m_declEntity) return XML_ERROR_NO_MEMORY; - if (declEntity->name != name) { + if (parser->m_declEntity->name != name) { poolDiscard(&dtd->pool); - declEntity = NULL; + parser->m_declEntity = NULL; } else { poolFinish(&dtd->pool); - declEntity->publicId = NULL; - declEntity->is_param = XML_FALSE; + parser->m_declEntity->publicId = NULL; + parser->m_declEntity->is_param = XML_FALSE; /* if we have a parent parser or are reading an internal parameter entity, then the entity declaration is not considered "internal" */ - declEntity->is_internal = !(parentParser || openInternalEntities); - if (entityDeclHandler) + parser->m_declEntity->is_internal = !(parser->m_parentParser || parser->m_openInternalEntities); + if (parser->m_entityDeclHandler) handleDefault = XML_FALSE; } } else { poolDiscard(&dtd->pool); - declEntity = NULL; + parser->m_declEntity = NULL; } } break; @@ -4894,90 +4875,90 @@ doProlog(XML_Parser parser, const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next); if (!name) return XML_ERROR_NO_MEMORY; - declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities, + parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, sizeof(ENTITY)); - if (!declEntity) + if (!parser->m_declEntity) return XML_ERROR_NO_MEMORY; - if (declEntity->name != name) { + if (parser->m_declEntity->name != name) { poolDiscard(&dtd->pool); - declEntity = NULL; + parser->m_declEntity = NULL; } else { poolFinish(&dtd->pool); - declEntity->publicId = NULL; - declEntity->is_param = XML_TRUE; + parser->m_declEntity->publicId = NULL; + parser->m_declEntity->is_param = XML_TRUE; /* if we have a parent parser or are reading an internal parameter entity, then the entity declaration is not considered "internal" */ - declEntity->is_internal = !(parentParser || openInternalEntities); - if (entityDeclHandler) + parser->m_declEntity->is_internal = !(parser->m_parentParser || parser->m_openInternalEntities); + if (parser->m_entityDeclHandler) handleDefault = XML_FALSE; } } else { poolDiscard(&dtd->pool); - declEntity = NULL; + parser->m_declEntity = NULL; } #else /* not XML_DTD */ - declEntity = NULL; + parser->m_declEntity = NULL; #endif /* XML_DTD */ break; case XML_ROLE_NOTATION_NAME: - declNotationPublicId = NULL; - declNotationName = NULL; - if (notationDeclHandler) { - declNotationName = poolStoreString(&tempPool, enc, s, next); - if (!declNotationName) + parser->m_declNotationPublicId = NULL; + parser->m_declNotationName = NULL; + if (parser->m_notationDeclHandler) { + parser->m_declNotationName = poolStoreString(&parser->m_tempPool, enc, s, next); + if (!parser->m_declNotationName) return XML_ERROR_NO_MEMORY; - poolFinish(&tempPool); + poolFinish(&parser->m_tempPool); handleDefault = XML_FALSE; } break; case XML_ROLE_NOTATION_PUBLIC_ID: if (!XmlIsPublicId(enc, s, next, eventPP)) return XML_ERROR_PUBLICID; - if (declNotationName) { /* means notationDeclHandler != NULL */ - XML_Char *tem = poolStoreString(&tempPool, + if (parser->m_declNotationName) { /* means m_notationDeclHandler != NULL */ + XML_Char *tem = poolStoreString(&parser->m_tempPool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!tem) return XML_ERROR_NO_MEMORY; normalizePublicId(tem); - declNotationPublicId = tem; - poolFinish(&tempPool); + parser->m_declNotationPublicId = tem; + poolFinish(&parser->m_tempPool); handleDefault = XML_FALSE; } break; case XML_ROLE_NOTATION_SYSTEM_ID: - if (declNotationName && notationDeclHandler) { + if (parser->m_declNotationName && parser->m_notationDeclHandler) { const XML_Char *systemId - = poolStoreString(&tempPool, enc, + = poolStoreString(&parser->m_tempPool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!systemId) return XML_ERROR_NO_MEMORY; *eventEndPP = s; - notationDeclHandler(handlerArg, - declNotationName, - curBase, + parser->m_notationDeclHandler(parser->m_handlerArg, + parser->m_declNotationName, + parser->m_curBase, systemId, - declNotationPublicId); + parser->m_declNotationPublicId); handleDefault = XML_FALSE; } - poolClear(&tempPool); + poolClear(&parser->m_tempPool); break; case XML_ROLE_NOTATION_NO_SYSTEM_ID: - if (declNotationPublicId && notationDeclHandler) { + if (parser->m_declNotationPublicId && parser->m_notationDeclHandler) { *eventEndPP = s; - notationDeclHandler(handlerArg, - declNotationName, - curBase, + parser->m_notationDeclHandler(parser->m_handlerArg, + parser->m_declNotationName, + parser->m_curBase, 0, - declNotationPublicId); + parser->m_declNotationPublicId); handleDefault = XML_FALSE; } - poolClear(&tempPool); + poolClear(&parser->m_tempPool); break; case XML_ROLE_ERROR: switch (tok) { @@ -4994,45 +4975,45 @@ doProlog(XML_Parser parser, case XML_ROLE_IGNORE_SECT: { enum XML_Error result; - if (defaultHandler) + if (parser->m_defaultHandler) reportDefault(parser, enc, s, next); handleDefault = XML_FALSE; result = doIgnoreSection(parser, enc, &next, end, nextPtr, haveMore); if (result != XML_ERROR_NONE) return result; else if (!next) { - processor = ignoreSectionProcessor; + parser->m_processor = ignoreSectionProcessor; return result; } } break; #endif /* XML_DTD */ case XML_ROLE_GROUP_OPEN: - if (prologState.level >= groupSize) { - if (groupSize) { - char *temp = (char *)REALLOC(groupConnector, groupSize *= 2); + if (parser->m_prologState.level >= parser->m_groupSize) { + if (parser->m_groupSize) { + char *temp = (char *)REALLOC(parser, parser->m_groupConnector, parser->m_groupSize *= 2); if (temp == NULL) { - groupSize /= 2; + parser->m_groupSize /= 2; return XML_ERROR_NO_MEMORY; } - groupConnector = temp; + parser->m_groupConnector = temp; if (dtd->scaffIndex) { - int *temp = (int *)REALLOC(dtd->scaffIndex, - groupSize * sizeof(int)); + int *temp = (int *)REALLOC(parser, dtd->scaffIndex, + parser->m_groupSize * sizeof(int)); if (temp == NULL) return XML_ERROR_NO_MEMORY; dtd->scaffIndex = temp; } } else { - groupConnector = (char *)MALLOC(groupSize = 32); - if (!groupConnector) { - groupSize = 0; + parser->m_groupConnector = (char *)MALLOC(parser, parser->m_groupSize = 32); + if (!parser->m_groupConnector) { + parser->m_groupSize = 0; return XML_ERROR_NO_MEMORY; } } } - groupConnector[prologState.level] = 0; + parser->m_groupConnector[parser->m_prologState.level] = 0; if (dtd->in_eldecl) { int myindex = nextScaffoldPart(parser); if (myindex < 0) @@ -5040,37 +5021,37 @@ doProlog(XML_Parser parser, dtd->scaffIndex[dtd->scaffLevel] = myindex; dtd->scaffLevel++; dtd->scaffold[myindex].type = XML_CTYPE_SEQ; - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; } break; case XML_ROLE_GROUP_SEQUENCE: - if (groupConnector[prologState.level] == ASCII_PIPE) + if (parser->m_groupConnector[parser->m_prologState.level] == ASCII_PIPE) return XML_ERROR_SYNTAX; - groupConnector[prologState.level] = ASCII_COMMA; - if (dtd->in_eldecl && elementDeclHandler) + parser->m_groupConnector[parser->m_prologState.level] = ASCII_COMMA; + if (dtd->in_eldecl && parser->m_elementDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_GROUP_CHOICE: - if (groupConnector[prologState.level] == ASCII_COMMA) + if (parser->m_groupConnector[parser->m_prologState.level] == ASCII_COMMA) return XML_ERROR_SYNTAX; if (dtd->in_eldecl - && !groupConnector[prologState.level] + && !parser->m_groupConnector[parser->m_prologState.level] && (dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type != XML_CTYPE_MIXED) ) { dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type = XML_CTYPE_CHOICE; - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; } - groupConnector[prologState.level] = ASCII_PIPE; + parser->m_groupConnector[parser->m_prologState.level] = ASCII_PIPE; break; case XML_ROLE_PARAM_ENTITY_REF: #ifdef XML_DTD case XML_ROLE_INNER_PARAM_ENTITY_REF: dtd->hasParamEntityRefs = XML_TRUE; - if (!paramEntityParsing) + if (!parser->m_paramEntityParsing) dtd->keepProcessing = dtd->standalone; else { const XML_Char *name; @@ -5086,9 +5067,9 @@ doProlog(XML_Parser parser, if yes, check that the entity exists, and that it is internal, otherwise call the skipped entity handler */ - if (prologState.documentEntity && + if (parser->m_prologState.documentEntity && (dtd->standalone - ? !openInternalEntities + ? !parser->m_openInternalEntities : !dtd->hasParamEntityRefs)) { if (!entity) return XML_ERROR_UNDEFINED_ENTITY; @@ -5119,8 +5100,8 @@ doProlog(XML_Parser parser, else if (!entity) { dtd->keepProcessing = dtd->standalone; /* cannot report skipped entities in declarations */ - if ((role == XML_ROLE_PARAM_ENTITY_REF) && skippedEntityHandler) { - skippedEntityHandler(handlerArg, name, 1); + if ((role == XML_ROLE_PARAM_ENTITY_REF) && parser->m_skippedEntityHandler) { + parser->m_skippedEntityHandler(parser->m_handlerArg, name, 1); handleDefault = XML_FALSE; } break; @@ -5137,10 +5118,10 @@ doProlog(XML_Parser parser, handleDefault = XML_FALSE; break; } - if (externalEntityRefHandler) { + if (parser->m_externalEntityRefHandler) { dtd->paramEntityRead = XML_FALSE; entity->open = XML_TRUE; - if (!externalEntityRefHandler(externalEntityRefHandlerArg, + if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg, 0, entity->base, entity->systemId, @@ -5162,17 +5143,17 @@ doProlog(XML_Parser parser, } #endif /* XML_DTD */ if (!dtd->standalone && - notStandaloneHandler && - !notStandaloneHandler(handlerArg)) + parser->m_notStandaloneHandler && + !parser->m_notStandaloneHandler(parser->m_handlerArg)) return XML_ERROR_NOT_STANDALONE; break; /* Element declaration stuff */ case XML_ROLE_ELEMENT_NAME: - if (elementDeclHandler) { - declElementType = getElementType(parser, enc, s, next); - if (!declElementType) + if (parser->m_elementDeclHandler) { + parser->m_declElementType = getElementType(parser, enc, s, next); + if (!parser->m_declElementType) return XML_ERROR_NO_MEMORY; dtd->scaffLevel = 0; dtd->scaffCount = 0; @@ -5184,8 +5165,8 @@ doProlog(XML_Parser parser, case XML_ROLE_CONTENT_ANY: case XML_ROLE_CONTENT_EMPTY: if (dtd->in_eldecl) { - if (elementDeclHandler) { - XML_Content * content = (XML_Content *) MALLOC(sizeof(XML_Content)); + if (parser->m_elementDeclHandler) { + XML_Content * content = (XML_Content *) MALLOC(parser, sizeof(XML_Content)); if (!content) return XML_ERROR_NO_MEMORY; content->quant = XML_CQUANT_NONE; @@ -5196,7 +5177,7 @@ doProlog(XML_Parser parser, XML_CTYPE_ANY : XML_CTYPE_EMPTY); *eventEndPP = s; - elementDeclHandler(handlerArg, declElementType->name, content); + parser->m_elementDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, content); handleDefault = XML_FALSE; } dtd->in_eldecl = XML_FALSE; @@ -5207,7 +5188,7 @@ doProlog(XML_Parser parser, if (dtd->in_eldecl) { dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type = XML_CTYPE_MIXED; - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; } break; @@ -5244,7 +5225,7 @@ doProlog(XML_Parser parser, nameLen = 0; for (; name[nameLen++]; ); dtd->contentStringLen += nameLen; - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; } break; @@ -5262,7 +5243,7 @@ doProlog(XML_Parser parser, quant = XML_CQUANT_PLUS; closeGroup: if (dtd->in_eldecl) { - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; dtd->scaffLevel--; dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel]].quant = quant; @@ -5272,7 +5253,7 @@ doProlog(XML_Parser parser, if (!model) return XML_ERROR_NO_MEMORY; *eventEndPP = s; - elementDeclHandler(handlerArg, declElementType->name, model); + parser->m_elementDeclHandler(parser->m_handlerArg, parser->m_declElementType->name, model); } dtd->in_eldecl = XML_FALSE; dtd->contentStringLen = 0; @@ -5299,31 +5280,31 @@ doProlog(XML_Parser parser, } break; case XML_ROLE_DOCTYPE_NONE: - if (startDoctypeDeclHandler) + if (parser->m_startDoctypeDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_ENTITY_NONE: - if (dtd->keepProcessing && entityDeclHandler) + if (dtd->keepProcessing && parser->m_entityDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_NOTATION_NONE: - if (notationDeclHandler) + if (parser->m_notationDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_ATTLIST_NONE: - if (dtd->keepProcessing && attlistDeclHandler) + if (dtd->keepProcessing && parser->m_attlistDeclHandler) handleDefault = XML_FALSE; break; case XML_ROLE_ELEMENT_NONE: - if (elementDeclHandler) + if (parser->m_elementDeclHandler) handleDefault = XML_FALSE; break; } /* end of big switch */ - if (handleDefault && defaultHandler) + if (handleDefault && parser->m_defaultHandler) reportDefault(parser, enc, s, next); - switch (ps_parsing) { + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: *nextPtr = next; return XML_ERROR_NONE; @@ -5343,18 +5324,18 @@ epilogProcessor(XML_Parser parser, const char *end, const char **nextPtr) { - processor = epilogProcessor; - eventPtr = s; + parser->m_processor = epilogProcessor; + parser->m_eventPtr = s; for (;;) { const char *next = NULL; - int tok = XmlPrologTok(encoding, s, end, &next); - eventEndPtr = next; + int tok = XmlPrologTok(parser->m_encoding, s, end, &next); + parser->m_eventEndPtr = next; switch (tok) { /* report partial linebreak - it might be the last token */ case -XML_TOK_PROLOG_S: - if (defaultHandler) { - reportDefault(parser, encoding, s, next); - if (ps_parsing == XML_FINISHED) + if (parser->m_defaultHandler) { + reportDefault(parser, parser->m_encoding, s, next); + if (parser->m_parsingStatus.parsing == XML_FINISHED) return XML_ERROR_ABORTED; } *nextPtr = next; @@ -5363,28 +5344,28 @@ epilogProcessor(XML_Parser parser, *nextPtr = s; return XML_ERROR_NONE; case XML_TOK_PROLOG_S: - if (defaultHandler) - reportDefault(parser, encoding, s, next); + if (parser->m_defaultHandler) + reportDefault(parser, parser->m_encoding, s, next); break; case XML_TOK_PI: - if (!reportProcessingInstruction(parser, encoding, s, next)) + if (!reportProcessingInstruction(parser, parser->m_encoding, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_COMMENT: - if (!reportComment(parser, encoding, s, next)) + if (!reportComment(parser, parser->m_encoding, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_INVALID: - eventPtr = next; + parser->m_eventPtr = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: - if (!ps_finalBuffer) { + if (!parser->m_parsingStatus.finalBuffer) { *nextPtr = s; return XML_ERROR_NONE; } @@ -5392,8 +5373,8 @@ epilogProcessor(XML_Parser parser, default: return XML_ERROR_JUNK_AFTER_DOC_ELEMENT; } - eventPtr = s = next; - switch (ps_parsing) { + parser->m_eventPtr = s = next; + switch (parser->m_parsingStatus.parsing) { case XML_SUSPENDED: *nextPtr = next; return XML_ERROR_NONE; @@ -5413,21 +5394,21 @@ processInternalEntity(XML_Parser parser, ENTITY *entity, enum XML_Error result; OPEN_INTERNAL_ENTITY *openEntity; - if (freeInternalEntities) { - openEntity = freeInternalEntities; - freeInternalEntities = openEntity->next; + if (parser->m_freeInternalEntities) { + openEntity = parser->m_freeInternalEntities; + parser->m_freeInternalEntities = openEntity->next; } else { - openEntity = (OPEN_INTERNAL_ENTITY *)MALLOC(sizeof(OPEN_INTERNAL_ENTITY)); + openEntity = (OPEN_INTERNAL_ENTITY *)MALLOC(parser, sizeof(OPEN_INTERNAL_ENTITY)); if (!openEntity) return XML_ERROR_NO_MEMORY; } entity->open = XML_TRUE; entity->processed = 0; - openEntity->next = openInternalEntities; - openInternalEntities = openEntity; + openEntity->next = parser->m_openInternalEntities; + parser->m_openInternalEntities = openEntity; openEntity->entity = entity; - openEntity->startTagLevel = tagLevel; + openEntity->startTagLevel = parser->m_tagLevel; openEntity->betweenDecl = betweenDecl; openEntity->internalEventPtr = NULL; openEntity->internalEventEndPtr = NULL; @@ -5438,26 +5419,26 @@ processInternalEntity(XML_Parser parser, ENTITY *entity, #ifdef XML_DTD if (entity->is_param) { - int tok = XmlPrologTok(internalEncoding, textStart, textEnd, &next); - result = doProlog(parser, internalEncoding, textStart, textEnd, tok, + int tok = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next); + result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd, tok, next, &next, XML_FALSE); } else #endif /* XML_DTD */ - result = doContent(parser, tagLevel, internalEncoding, textStart, + result = doContent(parser, parser->m_tagLevel, parser->m_internalEncoding, textStart, textEnd, &next, XML_FALSE); if (result == XML_ERROR_NONE) { - if (textEnd != next && ps_parsing == XML_SUSPENDED) { + if (textEnd != next && parser->m_parsingStatus.parsing == XML_SUSPENDED) { entity->processed = (int)(next - textStart); - processor = internalEntityProcessor; + parser->m_processor = internalEntityProcessor; } else { entity->open = XML_FALSE; - openInternalEntities = openEntity->next; + parser->m_openInternalEntities = openEntity->next; /* put openEntity back in list of free instances */ - openEntity->next = freeInternalEntities; - freeInternalEntities = openEntity; + openEntity->next = parser->m_freeInternalEntities; + parser->m_freeInternalEntities = openEntity; } } return result; @@ -5473,7 +5454,7 @@ internalEntityProcessor(XML_Parser parser, const char *textStart, *textEnd; const char *next; enum XML_Error result; - OPEN_INTERNAL_ENTITY *openEntity = openInternalEntities; + OPEN_INTERNAL_ENTITY *openEntity = parser->m_openInternalEntities; if (!openEntity) return XML_ERROR_UNEXPECTED_STATE; @@ -5485,44 +5466,44 @@ internalEntityProcessor(XML_Parser parser, #ifdef XML_DTD if (entity->is_param) { - int tok = XmlPrologTok(internalEncoding, textStart, textEnd, &next); - result = doProlog(parser, internalEncoding, textStart, textEnd, tok, + int tok = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next); + result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd, tok, next, &next, XML_FALSE); } else #endif /* XML_DTD */ - result = doContent(parser, openEntity->startTagLevel, internalEncoding, + result = doContent(parser, openEntity->startTagLevel, parser->m_internalEncoding, textStart, textEnd, &next, XML_FALSE); if (result != XML_ERROR_NONE) return result; - else if (textEnd != next && ps_parsing == XML_SUSPENDED) { + else if (textEnd != next && parser->m_parsingStatus.parsing == XML_SUSPENDED) { entity->processed = (int)(next - (char *)entity->textPtr); return result; } else { entity->open = XML_FALSE; - openInternalEntities = openEntity->next; + parser->m_openInternalEntities = openEntity->next; /* put openEntity back in list of free instances */ - openEntity->next = freeInternalEntities; - freeInternalEntities = openEntity; + openEntity->next = parser->m_freeInternalEntities; + parser->m_freeInternalEntities = openEntity; } #ifdef XML_DTD if (entity->is_param) { int tok; - processor = prologProcessor; - tok = XmlPrologTok(encoding, s, end, &next); - return doProlog(parser, encoding, s, end, tok, next, nextPtr, - (XML_Bool)!ps_finalBuffer); + parser->m_processor = prologProcessor; + tok = XmlPrologTok(parser->m_encoding, s, end, &next); + return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr, + (XML_Bool)!parser->m_parsingStatus.finalBuffer); } else #endif /* XML_DTD */ { - processor = contentProcessor; + parser->m_processor = contentProcessor; /* see externalEntityContentProcessor vs contentProcessor */ - return doContent(parser, parentParser ? 1 : 0, encoding, s, end, - nextPtr, (XML_Bool)!ps_finalBuffer); + return doContent(parser, parser->m_parentParser ? 1 : 0, parser->m_encoding, s, end, + nextPtr, (XML_Bool)!parser->m_parsingStatus.finalBuffer); } } @@ -5532,7 +5513,7 @@ errorProcessor(XML_Parser parser, const char *UNUSED_P(end), const char **UNUSED_P(nextPtr)) { - return errorCode; + return parser->m_errorCode; } static enum XML_Error @@ -5556,7 +5537,7 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, const char *ptr, const char *end, STRING_POOL *pool) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ for (;;) { const char *next; int tok = XmlAttributeValueTok(enc, ptr, end, &next); @@ -5564,12 +5545,12 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, case XML_TOK_NONE: return XML_ERROR_NONE; case XML_TOK_INVALID: - if (enc == encoding) - eventPtr = next; + if (enc == parser->m_encoding) + parser->m_eventPtr = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL: - if (enc == encoding) - eventPtr = ptr; + if (enc == parser->m_encoding) + parser->m_eventPtr = ptr; return XML_ERROR_INVALID_TOKEN; case XML_TOK_CHAR_REF: { @@ -5577,8 +5558,8 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, int i; int n = XmlCharRefNumber(enc, ptr); if (n < 0) { - if (enc == encoding) - eventPtr = ptr; + if (enc == parser->m_encoding) + parser->m_eventPtr = ptr; return XML_ERROR_BAD_CHAR_REF; } if (!isCdata @@ -5628,25 +5609,25 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, return XML_ERROR_NO_MEMORY; break; } - name = poolStoreString(&temp2Pool, enc, + name = poolStoreString(&parser->m_temp2Pool, enc, ptr + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!name) return XML_ERROR_NO_MEMORY; entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0); - poolDiscard(&temp2Pool); + poolDiscard(&parser->m_temp2Pool); /* First, determine if a check for an existing declaration is needed; if yes, check that the entity exists, and that it is internal. */ if (pool == &dtd->pool) /* are we called from prolog? */ checkEntityDecl = #ifdef XML_DTD - prologState.documentEntity && + parser->m_prologState.documentEntity && #endif /* XML_DTD */ (dtd->standalone - ? !openInternalEntities + ? !parser->m_openInternalEntities : !dtd->hasParamEntityRefs); - else /* if (pool == &tempPool): we are called from content */ + else /* if (pool == &parser->m_tempPool): we are called from content */ checkEntityDecl = !dtd->hasParamEntityRefs || dtd->standalone; if (checkEntityDecl) { if (!entity) @@ -5656,19 +5637,19 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, } else if (!entity) { /* Cannot report skipped entity here - see comments on - skippedEntityHandler. - if (skippedEntityHandler) - skippedEntityHandler(handlerArg, name, 0); + parser->m_skippedEntityHandler. + if (parser->m_skippedEntityHandler) + parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0); */ /* Cannot call the default handler because this would be out of sync with the call to the startElementHandler. - if ((pool == &tempPool) && defaultHandler) + if ((pool == &parser->m_tempPool) && parser->m_defaultHandler) reportDefault(parser, enc, ptr, next); */ break; } if (entity->open) { - if (enc == encoding) { + if (enc == parser->m_encoding) { /* It does not appear that this line can be executed. * * The "if (entity->open)" check catches recursive entity @@ -5686,25 +5667,25 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, * we keep the line and merely exclude it from coverage * tests. */ - eventPtr = ptr; /* LCOV_EXCL_LINE */ + parser->m_eventPtr = ptr; /* LCOV_EXCL_LINE */ } return XML_ERROR_RECURSIVE_ENTITY_REF; } if (entity->notation) { - if (enc == encoding) - eventPtr = ptr; + if (enc == parser->m_encoding) + parser->m_eventPtr = ptr; return XML_ERROR_BINARY_ENTITY_REF; } if (!entity->textPtr) { - if (enc == encoding) - eventPtr = ptr; + if (enc == parser->m_encoding) + parser->m_eventPtr = ptr; return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF; } else { enum XML_Error result; const XML_Char *textEnd = entity->textPtr + entity->textLen; entity->open = XML_TRUE; - result = appendAttributeValue(parser, internalEncoding, isCdata, + result = appendAttributeValue(parser, parser->m_internalEncoding, isCdata, (char *)entity->textPtr, (char *)textEnd, pool); entity->open = XML_FALSE; @@ -5725,8 +5706,8 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata, * * LCOV_EXCL_START */ - if (enc == encoding) - eventPtr = ptr; + if (enc == parser->m_encoding) + parser->m_eventPtr = ptr; return XML_ERROR_UNEXPECTED_STATE; /* LCOV_EXCL_STOP */ } @@ -5741,12 +5722,12 @@ storeEntityValue(XML_Parser parser, const char *entityTextPtr, const char *entityTextEnd) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ STRING_POOL *pool = &(dtd->entityValuePool); enum XML_Error result = XML_ERROR_NONE; #ifdef XML_DTD - int oldInEntityValue = prologState.inEntityValue; - prologState.inEntityValue = 1; + int oldInEntityValue = parser->m_prologState.inEntityValue; + parser->m_prologState.inEntityValue = 1; #endif /* XML_DTD */ /* never return Null for the value argument in EntityDeclHandler, since this would indicate an external entity; therefore we @@ -5762,10 +5743,10 @@ storeEntityValue(XML_Parser parser, switch (tok) { case XML_TOK_PARAM_ENTITY_REF: #ifdef XML_DTD - if (isParamEntity || enc != encoding) { + if (parser->m_isParamEntity || enc != parser->m_encoding) { const XML_Char *name; ENTITY *entity; - name = poolStoreString(&tempPool, enc, + name = poolStoreString(&parser->m_tempPool, enc, entityTextPtr + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!name) { @@ -5773,28 +5754,28 @@ storeEntityValue(XML_Parser parser, goto endEntityValue; } entity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, 0); - poolDiscard(&tempPool); + poolDiscard(&parser->m_tempPool); if (!entity) { /* not a well-formedness error - see XML 1.0: WFC Entity Declared */ /* cannot report skipped entity here - see comments on - skippedEntityHandler - if (skippedEntityHandler) - skippedEntityHandler(handlerArg, name, 0); + parser->m_skippedEntityHandler + if (parser->m_skippedEntityHandler) + parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0); */ dtd->keepProcessing = dtd->standalone; goto endEntityValue; } if (entity->open) { - if (enc == encoding) - eventPtr = entityTextPtr; + if (enc == parser->m_encoding) + parser->m_eventPtr = entityTextPtr; result = XML_ERROR_RECURSIVE_ENTITY_REF; goto endEntityValue; } if (entity->systemId) { - if (externalEntityRefHandler) { + if (parser->m_externalEntityRefHandler) { dtd->paramEntityRead = XML_FALSE; entity->open = XML_TRUE; - if (!externalEntityRefHandler(externalEntityRefHandlerArg, + if (!parser->m_externalEntityRefHandler(parser->m_externalEntityRefHandlerArg, 0, entity->base, entity->systemId, @@ -5813,7 +5794,7 @@ storeEntityValue(XML_Parser parser, else { entity->open = XML_TRUE; result = storeEntityValue(parser, - internalEncoding, + parser->m_internalEncoding, (char *)entity->textPtr, (char *)(entity->textPtr + entity->textLen)); @@ -5826,7 +5807,7 @@ storeEntityValue(XML_Parser parser, #endif /* XML_DTD */ /* In the internal subset, PE references are not legal within markup declarations, e.g entity values in this case. */ - eventPtr = entityTextPtr; + parser->m_eventPtr = entityTextPtr; result = XML_ERROR_PARAM_ENTITY_REF; goto endEntityValue; case XML_TOK_NONE: @@ -5855,8 +5836,8 @@ storeEntityValue(XML_Parser parser, int i; int n = XmlCharRefNumber(enc, entityTextPtr); if (n < 0) { - if (enc == encoding) - eventPtr = entityTextPtr; + if (enc == parser->m_encoding) + parser->m_eventPtr = entityTextPtr; result = XML_ERROR_BAD_CHAR_REF; goto endEntityValue; } @@ -5880,13 +5861,13 @@ storeEntityValue(XML_Parser parser, } break; case XML_TOK_PARTIAL: - if (enc == encoding) - eventPtr = entityTextPtr; + if (enc == parser->m_encoding) + parser->m_eventPtr = entityTextPtr; result = XML_ERROR_INVALID_TOKEN; goto endEntityValue; case XML_TOK_INVALID: - if (enc == encoding) - eventPtr = next; + if (enc == parser->m_encoding) + parser->m_eventPtr = next; result = XML_ERROR_INVALID_TOKEN; goto endEntityValue; default: @@ -5897,8 +5878,8 @@ storeEntityValue(XML_Parser parser, * * LCOV_EXCL_START */ - if (enc == encoding) - eventPtr = entityTextPtr; + if (enc == parser->m_encoding) + parser->m_eventPtr = entityTextPtr; result = XML_ERROR_UNEXPECTED_STATE; goto endEntityValue; /* LCOV_EXCL_STOP */ @@ -5907,7 +5888,7 @@ storeEntityValue(XML_Parser parser, } endEntityValue: #ifdef XML_DTD - prologState.inEntityValue = oldInEntityValue; + parser->m_prologState.inEntityValue = oldInEntityValue; #endif /* XML_DTD */ return result; } @@ -5942,25 +5923,25 @@ reportProcessingInstruction(XML_Parser parser, const ENCODING *enc, const XML_Char *target; XML_Char *data; const char *tem; - if (!processingInstructionHandler) { - if (defaultHandler) + if (!parser->m_processingInstructionHandler) { + if (parser->m_defaultHandler) reportDefault(parser, enc, start, end); return 1; } start += enc->minBytesPerChar * 2; tem = start + XmlNameLength(enc, start); - target = poolStoreString(&tempPool, enc, start, tem); + target = poolStoreString(&parser->m_tempPool, enc, start, tem); if (!target) return 0; - poolFinish(&tempPool); - data = poolStoreString(&tempPool, enc, + poolFinish(&parser->m_tempPool); + data = poolStoreString(&parser->m_tempPool, enc, XmlSkipS(enc, tem), end - enc->minBytesPerChar*2); if (!data) return 0; normalizeLines(data); - processingInstructionHandler(handlerArg, target, data); - poolClear(&tempPool); + parser->m_processingInstructionHandler(parser->m_handlerArg, target, data); + poolClear(&parser->m_tempPool); return 1; } @@ -5969,20 +5950,20 @@ reportComment(XML_Parser parser, const ENCODING *enc, const char *start, const char *end) { XML_Char *data; - if (!commentHandler) { - if (defaultHandler) + if (!parser->m_commentHandler) { + if (parser->m_defaultHandler) reportDefault(parser, enc, start, end); return 1; } - data = poolStoreString(&tempPool, + data = poolStoreString(&parser->m_tempPool, enc, start + enc->minBytesPerChar * 4, end - enc->minBytesPerChar * 3); if (!data) return 0; normalizeLines(data); - commentHandler(handlerArg, data); - poolClear(&tempPool); + parser->m_commentHandler(parser->m_handlerArg, data); + poolClear(&parser->m_tempPool); return 1; } @@ -5994,9 +5975,9 @@ reportDefault(XML_Parser parser, const ENCODING *enc, enum XML_Convert_Result convert_res; const char **eventPP; const char **eventEndPP; - if (enc == encoding) { - eventPP = &eventPtr; - eventEndPP = &eventEndPtr; + if (enc == parser->m_encoding) { + eventPP = &parser->m_eventPtr; + eventEndPP = &parser->m_eventEndPtr; } else { /* To get here, two things must be true; the parser must be @@ -6015,20 +5996,20 @@ reportDefault(XML_Parser parser, const ENCODING *enc, * * LCOV_EXCL_START */ - eventPP = &(openInternalEntities->internalEventPtr); - eventEndPP = &(openInternalEntities->internalEventEndPtr); + eventPP = &(parser->m_openInternalEntities->internalEventPtr); + eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr); /* LCOV_EXCL_STOP */ } do { - ICHAR *dataPtr = (ICHAR *)dataBuf; - convert_res = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd); + ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf; + convert_res = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd); *eventEndPP = s; - defaultHandler(handlerArg, dataBuf, (int)(dataPtr - (ICHAR *)dataBuf)); + parser->m_defaultHandler(parser->m_handlerArg, parser->m_dataBuf, (int)(dataPtr - (ICHAR *)parser->m_dataBuf)); *eventPP = s; } while ((convert_res != XML_CONVERT_COMPLETED) && (convert_res != XML_CONVERT_INPUT_INCOMPLETE)); } else - defaultHandler(handlerArg, (XML_Char *)s, (int)((XML_Char *)end - (XML_Char *)s)); + parser->m_defaultHandler(parser->m_handlerArg, (XML_Char *)s, (int)((XML_Char *)end - (XML_Char *)s)); } @@ -6050,16 +6031,18 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata, if (type->nDefaultAtts == type->allocDefaultAtts) { if (type->allocDefaultAtts == 0) { type->allocDefaultAtts = 8; - type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(type->allocDefaultAtts + type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(parser, type->allocDefaultAtts * sizeof(DEFAULT_ATTRIBUTE)); - if (!type->defaultAtts) + if (!type->defaultAtts) { + type->allocDefaultAtts = 0; return 0; + } } else { DEFAULT_ATTRIBUTE *temp; int count = type->allocDefaultAtts * 2; temp = (DEFAULT_ATTRIBUTE *) - REALLOC(type->defaultAtts, (count * sizeof(DEFAULT_ATTRIBUTE))); + REALLOC(parser, type->defaultAtts, (count * sizeof(DEFAULT_ATTRIBUTE))); if (temp == NULL) return 0; type->allocDefaultAtts = count; @@ -6079,7 +6062,7 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata, static int setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ const XML_Char *name; for (name = elementType->name; *name; name++) { if (*name == XML_T(ASCII_COLON)) { @@ -6110,7 +6093,7 @@ static ATTRIBUTE_ID * getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start, const char *end) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ ATTRIBUTE_ID *id; const XML_Char *name; if (!poolAppendChar(&dtd->pool, XML_T('\0'))) @@ -6127,7 +6110,7 @@ getAttributeId(XML_Parser parser, const ENCODING *enc, poolDiscard(&dtd->pool); else { poolFinish(&dtd->pool); - if (!ns) + if (!parser->m_ns) ; else if (name[0] == XML_T(ASCII_x) && name[1] == XML_T(ASCII_m) @@ -6174,20 +6157,20 @@ getAttributeId(XML_Parser parser, const ENCODING *enc, static const XML_Char * getContext(XML_Parser parser) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ HASH_TABLE_ITER iter; XML_Bool needSep = XML_FALSE; if (dtd->defaultPrefix.binding) { int i; int len; - if (!poolAppendChar(&tempPool, XML_T(ASCII_EQUALS))) + if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS))) return NULL; len = dtd->defaultPrefix.binding->uriLen; - if (namespaceSeparator) + if (parser->m_namespaceSeparator) len--; for (i = 0; i < len; i++) { - if (!poolAppendChar(&tempPool, dtd->defaultPrefix.binding->uri[i])) { + if (!poolAppendChar(&parser->m_tempPool, dtd->defaultPrefix.binding->uri[i])) { /* Because of memory caching, I don't believe this line can be * executed. * @@ -6230,18 +6213,18 @@ getContext(XML_Parser parser) */ continue; /* LCOV_EXCL_LINE */ } - if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP)) + if (needSep && !poolAppendChar(&parser->m_tempPool, CONTEXT_SEP)) return NULL; for (s = prefix->name; *s; s++) - if (!poolAppendChar(&tempPool, *s)) + if (!poolAppendChar(&parser->m_tempPool, *s)) return NULL; - if (!poolAppendChar(&tempPool, XML_T(ASCII_EQUALS))) + if (!poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS))) return NULL; len = prefix->binding->uriLen; - if (namespaceSeparator) + if (parser->m_namespaceSeparator) len--; for (i = 0; i < len; i++) - if (!poolAppendChar(&tempPool, prefix->binding->uri[i])) + if (!poolAppendChar(&parser->m_tempPool, prefix->binding->uri[i])) return NULL; needSep = XML_TRUE; } @@ -6255,73 +6238,73 @@ getContext(XML_Parser parser) break; if (!e->open) continue; - if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP)) + if (needSep && !poolAppendChar(&parser->m_tempPool, CONTEXT_SEP)) return NULL; for (s = e->name; *s; s++) - if (!poolAppendChar(&tempPool, *s)) + if (!poolAppendChar(&parser->m_tempPool, *s)) return 0; needSep = XML_TRUE; } - if (!poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return NULL; - return tempPool.start; + return parser->m_tempPool.start; } static XML_Bool setContext(XML_Parser parser, const XML_Char *context) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ const XML_Char *s = context; while (*context != XML_T('\0')) { if (*s == CONTEXT_SEP || *s == XML_T('\0')) { ENTITY *e; - if (!poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return XML_FALSE; - e = (ENTITY *)lookup(parser, &dtd->generalEntities, poolStart(&tempPool), 0); + e = (ENTITY *)lookup(parser, &dtd->generalEntities, poolStart(&parser->m_tempPool), 0); if (e) e->open = XML_TRUE; if (*s != XML_T('\0')) s++; context = s; - poolDiscard(&tempPool); + poolDiscard(&parser->m_tempPool); } else if (*s == XML_T(ASCII_EQUALS)) { PREFIX *prefix; - if (poolLength(&tempPool) == 0) + if (poolLength(&parser->m_tempPool) == 0) prefix = &dtd->defaultPrefix; else { - if (!poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return XML_FALSE; - prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&tempPool), + prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&parser->m_tempPool), sizeof(PREFIX)); if (!prefix) return XML_FALSE; - if (prefix->name == poolStart(&tempPool)) { + if (prefix->name == poolStart(&parser->m_tempPool)) { prefix->name = poolCopyString(&dtd->pool, prefix->name); if (!prefix->name) return XML_FALSE; } - poolDiscard(&tempPool); + poolDiscard(&parser->m_tempPool); } for (context = s + 1; *context != CONTEXT_SEP && *context != XML_T('\0'); context++) - if (!poolAppendChar(&tempPool, *context)) + if (!poolAppendChar(&parser->m_tempPool, *context)) return XML_FALSE; - if (!poolAppendChar(&tempPool, XML_T('\0'))) + if (!poolAppendChar(&parser->m_tempPool, XML_T('\0'))) return XML_FALSE; - if (addBinding(parser, prefix, NULL, poolStart(&tempPool), - &inheritedBindings) != XML_ERROR_NONE) + if (addBinding(parser, prefix, NULL, poolStart(&parser->m_tempPool), + &parser->m_inheritedBindings) != XML_ERROR_NONE) return XML_FALSE; - poolDiscard(&tempPool); + poolDiscard(&parser->m_tempPool); if (*context != XML_T('\0')) ++context; s = context; } else { - if (!poolAppendChar(&tempPool, *s)) + if (!poolAppendChar(&parser->m_tempPool, *s)) return XML_FALSE; s++; } @@ -6686,7 +6669,6 @@ hash(XML_Parser parser, KEY s) { struct siphash state; struct sipkey key; - (void)sip_tobin; (void)sip24_valid; copy_salt_to_sipkey(parser, &key); sip24_init(&state, &key); @@ -7006,8 +6988,8 @@ poolGrow(STRING_POOL *pool) int blockSize = (int)((unsigned)(pool->end - pool->start)*2U); size_t bytesToAllocate; - // NOTE: Needs to be calculated prior to calling `realloc` - // to avoid dangling pointers: + /* NOTE: Needs to be calculated prior to calling `realloc` + to avoid dangling pointers: */ const ptrdiff_t offsetInsideBlock = pool->ptr - pool->start; if (blockSize < 0) { @@ -7085,12 +7067,12 @@ poolGrow(STRING_POOL *pool) static int FASTCALL nextScaffoldPart(XML_Parser parser) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ CONTENT_SCAFFOLD * me; int next; if (!dtd->scaffIndex) { - dtd->scaffIndex = (int *)MALLOC(groupSize * sizeof(int)); + dtd->scaffIndex = (int *)MALLOC(parser, parser->m_groupSize * sizeof(int)); if (!dtd->scaffIndex) return -1; dtd->scaffIndex[0] = 0; @@ -7100,13 +7082,13 @@ nextScaffoldPart(XML_Parser parser) CONTENT_SCAFFOLD *temp; if (dtd->scaffold) { temp = (CONTENT_SCAFFOLD *) - REALLOC(dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD)); + REALLOC(parser, dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD)); if (temp == NULL) return -1; dtd->scaffSize *= 2; } else { - temp = (CONTENT_SCAFFOLD *)MALLOC(INIT_SCAFFOLD_ELEMENTS + temp = (CONTENT_SCAFFOLD *)MALLOC(parser, INIT_SCAFFOLD_ELEMENTS * sizeof(CONTENT_SCAFFOLD)); if (temp == NULL) return -1; @@ -7137,7 +7119,7 @@ build_node(XML_Parser parser, XML_Content **contpos, XML_Char **strpos) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ dest->type = dtd->scaffold[src_node].type; dest->quant = dtd->scaffold[src_node].quant; if (dest->type == XML_CTYPE_NAME) { @@ -7171,14 +7153,14 @@ build_node(XML_Parser parser, static XML_Content * build_model (XML_Parser parser) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ XML_Content *ret; XML_Content *cpos; XML_Char * str; int allocsize = (dtd->scaffCount * sizeof(XML_Content) + (dtd->contentStringLen * sizeof(XML_Char))); - ret = (XML_Content *)MALLOC(allocsize); + ret = (XML_Content *)MALLOC(parser, allocsize); if (!ret) return NULL; @@ -7195,7 +7177,7 @@ getElementType(XML_Parser parser, const char *ptr, const char *end) { - DTD * const dtd = _dtd; /* save one level of indirection */ + DTD * const dtd = parser->m_dtd; /* save one level of indirection */ const XML_Char *name = poolStoreString(&dtd->pool, enc, ptr, end); ELEMENT_TYPE *ret; diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c index 482df75..6371a35 100644 --- a/Modules/expat/xmltok.c +++ b/Modules/expat/xmltok.c @@ -30,8 +30,11 @@ USE OR OTHER DEALINGS IN THE SOFTWARE. */ +#if !defined(_WIN32) && defined(HAVE_EXPAT_CONFIG_H) +# include <pyconfig.h> +#endif #include <stddef.h> -#include <string.h> // memcpy +#include <string.h> /* memcpy */ #if defined(_MSC_VER) && (_MSC_VER <= 1700) /* for vs2012/11.0/1700 and earlier Visual Studio compilers */ @@ -66,7 +69,6 @@ { PREFIX(prologTok), PREFIX(contentTok), \ PREFIX(cdataSectionTok) IGNORE_SECTION_TOK_VTABLE }, \ { PREFIX(attributeValueTok), PREFIX(entityValueTok) }, \ - PREFIX(sameName), \ PREFIX(nameMatchesAscii), \ PREFIX(nameLength), \ PREFIX(skipS), \ @@ -363,7 +365,7 @@ enum { /* UTF8_cvalN is value of masked first byte of N byte sequence */ }; void -align_limit_to_full_utf8_characters(const char * from, const char ** fromLimRef) +_INTERNAL_trim_to_complete_utf8_characters(const char * from, const char ** fromLimRef) { const char * fromLim = *fromLimRef; size_t walked = 0; @@ -416,7 +418,7 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc), /* Avoid copying partial characters (from incomplete input). */ { const char * const fromLimBefore = fromLim; - align_limit_to_full_utf8_characters(*fromP, &fromLim); + _INTERNAL_trim_to_complete_utf8_characters(*fromP, &fromLim); if (fromLim < fromLimBefore) { input_incomplete = true; } @@ -424,12 +426,12 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc), { const ptrdiff_t bytesToCopy = fromLim - *fromP; - memcpy((void *)*toP, (const void *)*fromP, (size_t)bytesToCopy); + memcpy(*toP, *fromP, bytesToCopy); *fromP += bytesToCopy; *toP += bytesToCopy; } - if (output_exhausted) // needs to go first + if (output_exhausted) /* needs to go first */ return XML_CONVERT_OUTPUT_EXHAUSTED; else if (input_incomplete) return XML_CONVERT_INPUT_INCOMPLETE; @@ -1465,9 +1467,8 @@ unknown_toUtf8(const ENCODING *enc, return XML_CONVERT_OUTPUT_EXHAUSTED; (*fromP)++; } - do { - *(*toP)++ = *utf8++; - } while (--n != 0); + memcpy(*toP, utf8, n); + *toP += n; } } diff --git a/Modules/expat/xmltok.h b/Modules/expat/xmltok.h index 6d31879..50926f3 100644 --- a/Modules/expat/xmltok.h +++ b/Modules/expat/xmltok.h @@ -167,9 +167,6 @@ enum XML_Convert_Result { struct encoding { SCANNER scanners[XML_N_STATES]; SCANNER literalScanners[XML_N_LITERAL_TYPES]; - int (PTRCALL *sameName)(const ENCODING *, - const char *, - const char *); int (PTRCALL *nameMatchesAscii)(const ENCODING *, const char *, const char *, @@ -260,8 +257,6 @@ struct encoding { #define XmlEntityValueTok(enc, ptr, end, nextTokPtr) \ XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr) -#define XmlSameName(enc, ptr1, ptr2) (((enc)->sameName)(enc, ptr1, ptr2)) - #define XmlNameMatchesAscii(enc, ptr1, end1, ptr2) \ (((enc)->nameMatchesAscii)(enc, ptr1, end1, ptr2)) diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c index 93328b8..4d9ae7d 100644 --- a/Modules/expat/xmltok_impl.c +++ b/Modules/expat/xmltok_impl.c @@ -74,6 +74,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ @@ -102,6 +103,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ @@ -602,7 +604,7 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, return XML_TOK_INVALID; } } - /* fall through */ + /* fall through */ case BT_EQUALS: { int open; @@ -1442,6 +1444,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; + /* fall through */ default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ @@ -1653,87 +1656,14 @@ PREFIX(predefinedEntityName)(const ENCODING *UNUSED_P(enc), const char *ptr, return 0; } -/* This function does not appear to be called from anywhere within the - * library code. It is used via the macro XmlSameName(), which is - * defined but never used. Since it appears in the encoding function - * table, removing it is not a thing to be undertaken lightly. For - * the moment, we simply exclude it from coverage tests. - * - * LCOV_EXCL_START - */ -static int PTRCALL -PREFIX(sameName)(const ENCODING *enc, const char *ptr1, const char *ptr2) -{ - for (;;) { - switch (BYTE_TYPE(enc, ptr1)) { -#define LEAD_CASE(n) \ - case BT_LEAD ## n: \ - if (*ptr1++ != *ptr2++) \ - return 0; - LEAD_CASE(4) LEAD_CASE(3) LEAD_CASE(2) -#undef LEAD_CASE - /* fall through */ - if (*ptr1++ != *ptr2++) - return 0; - break; - case BT_NONASCII: - case BT_NMSTRT: -#ifdef XML_NS - case BT_COLON: -#endif - case BT_HEX: - case BT_DIGIT: - case BT_NAME: - case BT_MINUS: - if (*ptr2++ != *ptr1++) - return 0; - if (MINBPC(enc) > 1) { - if (*ptr2++ != *ptr1++) - return 0; - if (MINBPC(enc) > 2) { - if (*ptr2++ != *ptr1++) - return 0; - if (MINBPC(enc) > 3) { - if (*ptr2++ != *ptr1++) - return 0; - } - } - } - break; - default: - if (MINBPC(enc) == 1 && *ptr1 == *ptr2) - return 1; - switch (BYTE_TYPE(enc, ptr2)) { - case BT_LEAD2: - case BT_LEAD3: - case BT_LEAD4: - case BT_NONASCII: - case BT_NMSTRT: -#ifdef XML_NS - case BT_COLON: -#endif - case BT_HEX: - case BT_DIGIT: - case BT_NAME: - case BT_MINUS: - return 0; - default: - return 1; - } - } - } - /* not reached */ -} -/* LCOV_EXCL_STOP */ - static int PTRCALL PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1, const char *end1, const char *ptr2) { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (end1 - ptr1 < MINBPC(enc)) { - /* This line cannot be executed. THe incoming data has already - * been tokenized once, so imcomplete characters like this have + /* This line cannot be executed. The incoming data has already + * been tokenized once, so incomplete characters like this have * already been eliminated from the input. Retaining the * paranoia check is still valuable, however. */ diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c index 916e481..700c1a3 100644 --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -771,10 +771,8 @@ debug_cycle(char *msg, PyObject *op) * garbage list (a Python list), else only the objects in finalizers with * __del__ methods are appended to garbage. All objects in finalizers are * merged into the old list regardless. - * Returns 0 if all OK, <0 on error (out of memory to grow the garbage list). - * The finalizers list is made empty on a successful return. */ -static int +static void handle_finalizers(PyGC_Head *finalizers, PyGC_Head *old) { PyGC_Head *gc = finalizers->gc.gc_next; @@ -789,12 +787,11 @@ handle_finalizers(PyGC_Head *finalizers, PyGC_Head *old) if ((debug & DEBUG_SAVEALL) || has_finalizer(op)) { if (PyList_Append(garbage, op) < 0) - return -1; + break; } } gc_list_merge(finalizers, old); - return 0; } /* Break reference cycles by clearing the containers involved. This is @@ -1012,7 +1009,7 @@ collect(int generation) * reachable list of garbage. The programmer has to deal with * this if they insist on creating this type of structure. */ - (void)handle_finalizers(&finalizers, old); + handle_finalizers(&finalizers, old); /* Clear free list only during the collection of the highest * generation */ @@ -1436,8 +1433,11 @@ PyGC_Collect(void) if (collecting) n = 0; /* already collecting, don't do anything */ else { + PyObject *exc, *value, *tb; collecting = 1; + PyErr_Fetch(&exc, &value, &tb); n = collect(NUM_GENERATIONS - 1); + PyErr_Restore(exc, value, tb); collecting = 0; } @@ -1539,6 +1539,7 @@ _PyObject_GC_Resize(PyVarObject *op, Py_ssize_t nitems) { const size_t basicsize = _PyObject_VAR_SIZE(Py_TYPE(op), nitems); PyGC_Head *g = AS_GC(op); + assert(!IS_TRACKED(op)); if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head)) return (PyVarObject *)PyErr_NoMemory(); g = (PyGC_Head *)PyObject_REALLOC(g, sizeof(PyGC_Head) + basicsize); diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 01ed36b..67354a7 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -1188,7 +1188,7 @@ math_ldexp(PyObject *self, PyObject *args) if (! PyArg_ParseTuple(args, "dO:ldexp", &x, &oexp)) return NULL; - if (PyLong_Check(oexp) || PyInt_Check(oexp)) { + if (_PyAnyInt_Check(oexp)) { /* on overflow, replace exponent with either LONG_MAX or LONG_MIN, depending on the sign. */ exp = PyLong_AsLongAndOverflow(oexp, &overflow); diff --git a/Modules/md5module.c b/Modules/md5module.c index 103da14..dd90a22 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -236,7 +236,7 @@ copy() -- return a copy of the current md5 object"); static PyTypeObject MD5type = { PyVarObject_HEAD_INIT(NULL, 0) "_md5.md5", /*tp_name*/ - sizeof(md5object), /*tp_size*/ + sizeof(md5object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)md5_dealloc, /*tp_dealloc*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index 2ae52c7..02b31ca 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -818,24 +818,6 @@ mmap_subscript(mmap_object *self, PyObject *item) } } -static PyObject * -mmap_concat(mmap_object *self, PyObject *bb) -{ - CHECK_VALID(NULL); - PyErr_SetString(PyExc_SystemError, - "mmaps don't support concatenation"); - return NULL; -} - -static PyObject * -mmap_repeat(mmap_object *self, Py_ssize_t n) -{ - CHECK_VALID(NULL); - PyErr_SetString(PyExc_SystemError, - "mmaps don't support repeat operation"); - return NULL; -} - static int mmap_ass_slice(mmap_object *self, Py_ssize_t ilow, Py_ssize_t ihigh, PyObject *v) { @@ -993,9 +975,9 @@ mmap_ass_subscript(mmap_object *self, PyObject *item, PyObject *value) static PySequenceMethods mmap_as_sequence = { (lenfunc)mmap_length, /*sq_length*/ - (binaryfunc)mmap_concat, /*sq_concat*/ - (ssizeargfunc)mmap_repeat, /*sq_repeat*/ - (ssizeargfunc)mmap_item, /*sq_item*/ + 0, /*sq_concat*/ + 0, /*sq_repeat*/ + (ssizeargfunc)mmap_item, /*sq_item*/ (ssizessizeargfunc)mmap_slice, /*sq_slice*/ (ssizeobjargproc)mmap_ass_item, /*sq_ass_item*/ (ssizessizeobjargproc)mmap_ass_slice, /*sq_ass_slice*/ @@ -1044,7 +1026,7 @@ To map anonymous memory, pass -1 as the fileno (both versions)."); static PyTypeObject mmap_object_type = { PyVarObject_HEAD_INIT(NULL, 0) "mmap.mmap", /* tp_name */ - sizeof(mmap_object), /* tp_size */ + sizeof(mmap_object), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) mmap_object_dealloc, /* tp_dealloc */ diff --git a/Modules/nismodule.c b/Modules/nismodule.c index 6acab63..e8caea9 100644 --- a/Modules/nismodule.c +++ b/Modules/nismodule.c @@ -405,6 +405,7 @@ nis_maps (PyObject *self, PyObject *args, PyObject *kwdict) PyObject *str = PyString_FromString(maps->map); if (!str || PyList_Append(list, str) < 0) { + Py_XDECREF(str); Py_DECREF(list); list = NULL; break; diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 284cc61..f833bb2 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -842,7 +842,7 @@ oss_mixer_getattr(oss_mixer_t *self, char *name) static PyTypeObject OSSAudioType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_audio_device", /*tp_name*/ - sizeof(oss_audio_t), /*tp_size*/ + sizeof(oss_audio_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_dealloc, /*tp_dealloc*/ @@ -856,7 +856,7 @@ static PyTypeObject OSSAudioType = { static PyTypeObject OSSMixerType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_mixer_device", /*tp_name*/ - sizeof(oss_mixer_t), /*tp_size*/ + sizeof(oss_mixer_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_mixer_dealloc, /*tp_dealloc*/ diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c index 7955542..fcc618d 100644 --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -121,10 +121,10 @@ node2tuple(node *n, /* node to convert */ if (result != NULL) { (void) addelem(result, 0, PyInt_FromLong(TYPE(n))); (void) addelem(result, 1, PyString_FromString(STR(n))); - if (lineno == 1) + if (lineno) (void) addelem(result, 2, PyInt_FromLong(n->n_lineno)); - if (col_offset == 1) - (void) addelem(result, 3, PyInt_FromLong(n->n_col_offset)); + if (col_offset) + (void) addelem(result, 2 + lineno, PyInt_FromLong(n->n_col_offset)); } return (result); } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 2baf920..7a1a694 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -6121,7 +6121,7 @@ posix_setgroups(PyObject *self, PyObject *groups) elem = PySequence_GetItem(groups, i); if (!elem) return NULL; - if (!PyInt_Check(elem) && !PyLong_Check(elem)) { + if (!_PyAnyInt_Check(elem)) { PyErr_SetString(PyExc_TypeError, "groups must be integers"); Py_DECREF(elem); diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index 2b4d312..a1a870d 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -398,8 +398,10 @@ string_intern(xmlparseobject *self, const char* str) if (!value) { if (PyDict_SetItem(self->intern, result, result) == 0) return result; - else + else { + Py_DECREF(result); return NULL; + } } Py_INCREF(value); Py_DECREF(result); @@ -547,6 +549,7 @@ my_StartElementHandler(void *userData, flag_error(self); Py_DECREF(n); Py_DECREF(v); + Py_DECREF(container); return; } else { @@ -555,12 +558,14 @@ my_StartElementHandler(void *userData, } } args = string_intern(self, name); - if (args != NULL) - args = Py_BuildValue("(NN)", args, container); if (args == NULL) { Py_DECREF(container); return; } + args = Py_BuildValue("(NN)", args, container); + if (args == NULL) { + return; + } /* Container is now a borrowed reference; ignore it. */ self->in_callback = 1; rv = call_with_frame(getcode(StartElement, "StartElement", __LINE__), @@ -742,7 +747,6 @@ my_ElementDeclHandler(void *userData, } args = Py_BuildValue("NN", nameobj, modelobj); if (args == NULL) { - Py_DECREF(modelobj); flag_error(self); goto finally; } @@ -1080,21 +1084,7 @@ xmlparse_ExternalEntityParserCreate(xmlparseobject *self, PyObject *args) return NULL; new_parser->buffer_size = self->buffer_size; new_parser->buffer_used = 0; - if (self->buffer != NULL) { - new_parser->buffer = malloc(new_parser->buffer_size); - if (new_parser->buffer == NULL) { -#ifndef Py_TPFLAGS_HAVE_GC - /* Code for versions 2.0 and 2.1 */ - PyObject_Del(new_parser); -#else - /* Code for versions 2.2 and later. */ - PyObject_GC_Del(new_parser); -#endif - return PyErr_NoMemory(); - } - } - else - new_parser->buffer = NULL; + new_parser->buffer = NULL; new_parser->returns_unicode = self->returns_unicode; new_parser->ordered_attributes = self->ordered_attributes; new_parser->specified_attributes = self->specified_attributes; @@ -1116,6 +1106,14 @@ xmlparse_ExternalEntityParserCreate(xmlparseobject *self, PyObject *args) return PyErr_NoMemory(); } + if (self->buffer != NULL) { + new_parser->buffer = malloc(new_parser->buffer_size); + if (new_parser->buffer == NULL) { + Py_DECREF(new_parser); + return PyErr_NoMemory(); + } + } + XML_SetUserData(new_parser->itself, (void *)new_parser); /* allocate and clear handlers first */ @@ -1306,6 +1304,8 @@ newxmlparseobject(char *encoding, char *namespace_separator, PyObject *intern) self->in_callback = 0; self->ns_prefixes = 0; self->handlers = NULL; + self->intern = intern; + Py_XINCREF(self->intern); if (namespace_separator != NULL) { self->itself = XML_ParserCreateNS(encoding, *namespace_separator); } @@ -1323,8 +1323,6 @@ newxmlparseobject(char *encoding, char *namespace_separator, PyObject *intern) XML_SetHashSalt(self->itself, (unsigned long)_Py_HashSecret.prefix); #endif - self->intern = intern; - Py_XINCREF(self->intern); #ifdef Py_TPFLAGS_HAVE_GC PyObject_GC_Track(self); #else @@ -2042,6 +2040,11 @@ MODULE_INITFUNC(void) capi.SetProcessingInstructionHandler = XML_SetProcessingInstructionHandler; capi.SetUnknownEncodingHandler = XML_SetUnknownEncodingHandler; capi.SetUserData = XML_SetUserData; +#if XML_COMBINED_VERSION >= 20100 + capi.SetHashSalt = XML_SetHashSalt; +#else + capi.SetHashSalt = NULL; +#endif /* export using capsule */ capi_object = PyCapsule_New(&capi, PyExpat_CAPSULE_NAME, NULL); diff --git a/Modules/readline.c b/Modules/readline.c index 1e10dd7..0262135 100644 --- a/Modules/readline.c +++ b/Modules/readline.c @@ -792,8 +792,7 @@ on_completion_display_matches_hook(char **matches, s = PyString_FromString(matches[i+1]); if (s == NULL) goto error; - if (PyList_SetItem(m, i, s) == -1) - goto error; + PyList_SET_ITEM(m, i, s); } r = PyObject_CallFunction(completion_display_matches_hook, diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index 1dec6a1..dc22f10 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -601,10 +601,7 @@ poll_poll(pollObject *self, PyObject *args) goto error; } PyTuple_SET_ITEM(value, 1, num); - if ((PyList_SetItem(result_list, j, value)) == -1) { - Py_DECREF(value); - goto error; - } + PyList_SET_ITEM(result_list, j, value); i++; } } diff --git a/Modules/sha256module.c b/Modules/sha256module.c index 8f18faa..76e5220 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -537,7 +537,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA224type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha224", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ @@ -571,7 +571,7 @@ static PyTypeObject SHA224type = { static PyTypeObject SHA256type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha256", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index abc9021..e3733e0 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -603,7 +603,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA384type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha384", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ @@ -637,7 +637,7 @@ static PyTypeObject SHA384type = { static PyTypeObject SHA512type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha512", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index ef70d10..8628f7a 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -95,13 +95,6 @@ static PyObject *DefaultHandler; static PyObject *IgnoreHandler; static PyObject *IntHandler; -/* On Solaris 8, gcc will produce a warning that the function - declaration is not a prototype. This is caused by the definition of - SIG_DFL as (void (*)())0; the correct declaration would have been - (void (*)(int))0. */ - -static PyOS_sighandler_t old_siginthandler = SIG_DFL; - #ifdef HAVE_GETITIMER static PyObject *ItimerError; @@ -629,7 +622,7 @@ initsignal(void) /* Install default int handler */ Py_INCREF(IntHandler); Py_SETREF(Handlers[SIGINT].func, IntHandler); - old_siginthandler = PyOS_setsig(SIGINT, signal_handler); + PyOS_setsig(SIGINT, signal_handler); } #ifdef SIGHUP @@ -872,14 +865,11 @@ finisignal(void) int i; PyObject *func; - PyOS_setsig(SIGINT, old_siginthandler); - old_siginthandler = SIG_DFL; - for (i = 1; i < NSIG; i++) { func = Handlers[i].func; Handlers[i].tripped = 0; Handlers[i].func = NULL; - if (i != SIGINT && func != NULL && func != Py_None && + if (func != NULL && func != Py_None && func != DefaultHandler && func != IgnoreHandler) PyOS_setsig(i, SIG_DFL); Py_XDECREF(func); diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 8d36705..0139754 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -4194,7 +4194,7 @@ socket_getaddrinfo(PyObject *self, PyObject *args) "getaddrinfo() argument 1 must be string or None"); return NULL; } - if (PyInt_Check(pobj) || PyLong_Check(pobj)) { + if (_PyAnyInt_Check(pobj)) { long value = PyLong_AsLong(pobj); if (value == -1 && PyErr_Occurred()) return NULL; @@ -4249,9 +4249,11 @@ socket_getaddrinfo(PyObject *self, PyObject *args) if (single == NULL) goto err; - if (PyList_Append(all, single)) + if (PyList_Append(all, single)) { + Py_DECREF(single); goto err; - Py_XDECREF(single); + } + Py_DECREF(single); } Py_XDECREF(idna); if (res0) diff --git a/Modules/svmodule.c b/Modules/svmodule.c index 42c49c8..14f236c 100644 --- a/Modules/svmodule.c +++ b/Modules/svmodule.c @@ -686,7 +686,7 @@ sv_LoadMap(svobject *self, PyObject *args) if (!cell) goto finally; - if (!PyInt_Check(cell) && !PyLong_Check(cell)) { + if (!_PyAnyInt_Check(cell)) { PyErr_BadArgument(); goto finally; } @@ -757,7 +757,7 @@ doParams(svobject *self, PyObject *args, if (!v) goto finally; - if (!PyInt_Check(v) && !PyLong_Check(v)) { + if (!_PyAnyInt_Check(v)) { PyErr_BadArgument(); goto finally; } diff --git a/Modules/termios.c b/Modules/termios.c index 9d4d780..4349df6 100644 --- a/Modules/termios.c +++ b/Modules/termios.c @@ -120,11 +120,11 @@ termios_tcgetattr(PyObject *self, PyObject *args) PyList_SetItem(v, 3, PyInt_FromLong((long)mode.c_lflag)); PyList_SetItem(v, 4, PyInt_FromLong((long)ispeed)); PyList_SetItem(v, 5, PyInt_FromLong((long)ospeed)); - PyList_SetItem(v, 6, cc); - if (PyErr_Occurred()){ + if (PyErr_Occurred()) { Py_DECREF(v); goto err; } + PyList_SetItem(v, 6, cc); return v; err: Py_DECREF(cc); @@ -185,7 +185,7 @@ termios_tcsetattr(PyObject *self, PyObject *args) if (PyString_Check(v) && PyString_Size(v) == 1) mode.c_cc[i] = (cc_t) * PyString_AsString(v); - else if (PyInt_Check(v) || PyLong_Check(v)) { + else if (_PyAnyInt_Check(v)) { mode.c_cc[i] = (cc_t) PyInt_AsLong(v); if (mode.c_cc[i] == (cc_t) -1 && PyErr_Occurred()) return NULL; diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c index 6b01fc7..df6ffe3 100644 --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -664,14 +664,18 @@ nfc_nfkc(PyObject *self, PyObject *input, int k) pairs, since we always have decomposed data. */ if (LBase <= *i && *i < (LBase+LCount) && i + 1 < end && - VBase <= i[1] && i[1] <= (VBase+VCount)) { + VBase <= i[1] && i[1] < (VBase+VCount)) { + /* check L character is a modern leading consonant (0x1100 ~ 0x1112) + and V character is a modern vowel (0x1161 ~ 0x1175). */ int LIndex, VIndex; LIndex = i[0] - LBase; VIndex = i[1] - VBase; code = SBase + (LIndex*VCount+VIndex)*TCount; i+=2; if (i < end && - TBase <= *i && *i <= (TBase+TCount)) { + TBase < *i && *i < (TBase+TCount)) { + /* check T character is a modern trailing consonant + (0x11A8 ~ 0x11C2). */ code += *i-TBase; i++; } diff --git a/Objects/abstract.c b/Objects/abstract.c index aa92ea9..75c1a10 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -1492,14 +1492,13 @@ PyNumber_Index(PyObject *item) PyObject *result = NULL; if (item == NULL) return null_error(); - if (PyInt_Check(item) || PyLong_Check(item)) { + if (_PyAnyInt_Check(item)) { Py_INCREF(item); return item; } if (PyIndex_Check(item)) { result = item->ob_type->tp_as_number->nb_index(item); - if (result && - !PyInt_Check(result) && !PyLong_Check(result)) { + if (result && !_PyAnyInt_Check(result)) { PyErr_Format(PyExc_TypeError, "__index__ returned non-(int,long) " \ "(type %.200s)", @@ -1574,8 +1573,7 @@ _PyNumber_ConvertIntegralToInt(PyObject *integral, const char* error_format) return NULL; } - if (integral && (!PyInt_Check(integral) && - !PyLong_Check(integral))) { + if (integral && !_PyAnyInt_Check(integral)) { /* Don't go through tp_as_number->nb_int to avoid hitting the classic class fallback to __trunc__. */ PyObject *int_func = PyObject_GetAttr(integral, int_name); @@ -1586,8 +1584,7 @@ _PyNumber_ConvertIntegralToInt(PyObject *integral, const char* error_format) Py_DECREF(integral); integral = PyEval_CallObject(int_func, NULL); Py_DECREF(int_func); - if (integral && (!PyInt_Check(integral) && - !PyLong_Check(integral))) { + if (integral && !_PyAnyInt_Check(integral)) { goto non_integral_error; } } @@ -1632,7 +1629,7 @@ PyNumber_Int(PyObject *o) if (m && m->nb_int) { /* This should include subclasses of int */ /* Classic classes always take this branch. */ PyObject *res = m->nb_int(o); - if (res && (!PyInt_Check(res) && !PyLong_Check(res))) { + if (res && !_PyAnyInt_Check(res)) { PyErr_Format(PyExc_TypeError, "__int__ returned non-int (type %.200s)", res->ob_type->tp_name); diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index 04c2506..a96d6d9 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -35,14 +35,16 @@ _getbytevalue(PyObject* arg, int *value) *value = Py_CHARMASK(((PyBytesObject*)arg)->ob_sval[0]); return 1; } - else if (PyInt_Check(arg) || PyLong_Check(arg)) { + else if (_PyAnyInt_Check(arg)) { face_value = PyLong_AsLong(arg); } else { PyObject *index = PyNumber_Index(arg); if (index == NULL) { - PyErr_Format(PyExc_TypeError, - "an integer or string of size 1 is required"); + if (PyErr_ExceptionMatches(PyExc_TypeError)) { + PyErr_Format(PyExc_TypeError, + "an integer or string of size 1 is required"); + } return 0; } face_value = PyLong_AsLong(index); @@ -852,7 +854,7 @@ bytearray_init(PyByteArrayObject *self, PyObject *args, PyObject *kwds) /* Is it an int? */ count = PyNumber_AsSsize_t(arg, PyExc_OverflowError); if (count == -1 && PyErr_Occurred()) { - if (PyErr_ExceptionMatches(PyExc_OverflowError)) + if (!PyErr_ExceptionMatches(PyExc_TypeError)) return -1; PyErr_Clear(); } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c deleted file mode 100644 index 7ba90aa..0000000 --- a/Objects/bytesobject.c +++ /dev/null @@ -1,3432 +0,0 @@ -/* bytes object implementation */ - -#define PY_SSIZE_T_CLEAN - -#include "Python.h" -#include "internal/mem.h" -#include "internal/pystate.h" - -#include "bytes_methods.h" -#include "pystrhex.h" -#include <stddef.h> - -/*[clinic input] -class bytes "PyBytesObject *" "&PyBytes_Type" -[clinic start generated code]*/ -/*[clinic end generated code: output=da39a3ee5e6b4b0d input=7a238f965d64892b]*/ - -#include "clinic/bytesobject.c.h" - -#ifdef COUNT_ALLOCS -Py_ssize_t null_strings, one_strings; -#endif - -static PyBytesObject *characters[UCHAR_MAX + 1]; -static PyBytesObject *nullstring; - -/* PyBytesObject_SIZE gives the basic size of a string; any memory allocation - for a string of length n should request PyBytesObject_SIZE + n bytes. - - Using PyBytesObject_SIZE instead of sizeof(PyBytesObject) saves - 3 bytes per string allocation on a typical system. -*/ -#define PyBytesObject_SIZE (offsetof(PyBytesObject, ob_sval) + 1) - -/* Forward declaration */ -Py_LOCAL_INLINE(Py_ssize_t) _PyBytesWriter_GetSize(_PyBytesWriter *writer, - char *str); - -/* - For PyBytes_FromString(), the parameter `str' points to a null-terminated - string containing exactly `size' bytes. - - For PyBytes_FromStringAndSize(), the parameter `str' is - either NULL or else points to a string containing at least `size' bytes. - For PyBytes_FromStringAndSize(), the string in the `str' parameter does - not have to be null-terminated. (Therefore it is safe to construct a - substring by calling `PyBytes_FromStringAndSize(origstring, substrlen)'.) - If `str' is NULL then PyBytes_FromStringAndSize() will allocate `size+1' - bytes (setting the last byte to the null terminating character) and you can - fill in the data yourself. If `str' is non-NULL then the resulting - PyBytes object must be treated as immutable and you must not fill in nor - alter the data yourself, since the strings may be shared. - - The PyObject member `op->ob_size', which denotes the number of "extra - items" in a variable-size object, will contain the number of bytes - allocated for string data, not counting the null terminating character. - It is therefore equal to the `size' parameter (for - PyBytes_FromStringAndSize()) or the length of the string in the `str' - parameter (for PyBytes_FromString()). -*/ -static PyObject * -_PyBytes_FromSize(Py_ssize_t size, int use_calloc) -{ - PyBytesObject *op; - assert(size >= 0); - - if (size == 0 && (op = nullstring) != NULL) { -#ifdef COUNT_ALLOCS - null_strings++; -#endif - Py_INCREF(op); - return (PyObject *)op; - } - - if ((size_t)size > (size_t)PY_SSIZE_T_MAX - PyBytesObject_SIZE) { - PyErr_SetString(PyExc_OverflowError, - "byte string is too large"); - return NULL; - } - - /* Inline PyObject_NewVar */ - if (use_calloc) - op = (PyBytesObject *)PyObject_Calloc(1, PyBytesObject_SIZE + size); - else - op = (PyBytesObject *)PyObject_Malloc(PyBytesObject_SIZE + size); - if (op == NULL) - return PyErr_NoMemory(); - (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); - op->ob_shash = -1; - if (!use_calloc) - op->ob_sval[size] = '\0'; - /* empty byte string singleton */ - if (size == 0) { - nullstring = op; - Py_INCREF(op); - } - return (PyObject *) op; -} - -PyObject * -PyBytes_FromStringAndSize(const char *str, Py_ssize_t size) -{ - PyBytesObject *op; - if (size < 0) { - PyErr_SetString(PyExc_SystemError, - "Negative size passed to PyBytes_FromStringAndSize"); - return NULL; - } - if (size == 1 && str != NULL && - (op = characters[*str & UCHAR_MAX]) != NULL) - { -#ifdef COUNT_ALLOCS - one_strings++; -#endif - Py_INCREF(op); - return (PyObject *)op; - } - - op = (PyBytesObject *)_PyBytes_FromSize(size, 0); - if (op == NULL) - return NULL; - if (str == NULL) - return (PyObject *) op; - - memcpy(op->ob_sval, str, size); - /* share short strings */ - if (size == 1) { - characters[*str & UCHAR_MAX] = op; - Py_INCREF(op); - } - return (PyObject *) op; -} - -PyObject * -PyBytes_FromString(const char *str) -{ - size_t size; - PyBytesObject *op; - - assert(str != NULL); - size = strlen(str); - if (size > PY_SSIZE_T_MAX - PyBytesObject_SIZE) { - PyErr_SetString(PyExc_OverflowError, - "byte string is too long"); - return NULL; - } - if (size == 0 && (op = nullstring) != NULL) { -#ifdef COUNT_ALLOCS - null_strings++; -#endif - Py_INCREF(op); - return (PyObject *)op; - } - if (size == 1 && (op = characters[*str & UCHAR_MAX]) != NULL) { -#ifdef COUNT_ALLOCS - one_strings++; -#endif - Py_INCREF(op); - return (PyObject *)op; - } - - /* Inline PyObject_NewVar */ - op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + size); - if (op == NULL) - return PyErr_NoMemory(); - (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); - op->ob_shash = -1; - memcpy(op->ob_sval, str, size+1); - /* share short strings */ - if (size == 0) { - nullstring = op; - Py_INCREF(op); - } else if (size == 1) { - characters[*str & UCHAR_MAX] = op; - Py_INCREF(op); - } - return (PyObject *) op; -} - -PyObject * -PyBytes_FromFormatV(const char *format, va_list vargs) -{ - char *s; - const char *f; - const char *p; - Py_ssize_t prec; - int longflag; - int size_tflag; - /* Longest 64-bit formatted numbers: - - "18446744073709551615\0" (21 bytes) - - "-9223372036854775808\0" (21 bytes) - Decimal takes the most space (it isn't enough for octal.) - - Longest 64-bit pointer representation: - "0xffffffffffffffff\0" (19 bytes). */ - char buffer[21]; - _PyBytesWriter writer; - - _PyBytesWriter_Init(&writer); - - s = _PyBytesWriter_Alloc(&writer, strlen(format)); - if (s == NULL) - return NULL; - writer.overallocate = 1; - -#define WRITE_BYTES(str) \ - do { \ - s = _PyBytesWriter_WriteBytes(&writer, s, (str), strlen(str)); \ - if (s == NULL) \ - goto error; \ - } while (0) - - for (f = format; *f; f++) { - if (*f != '%') { - *s++ = *f; - continue; - } - - p = f++; - - /* ignore the width (ex: 10 in "%10s") */ - while (Py_ISDIGIT(*f)) - f++; - - /* parse the precision (ex: 10 in "%.10s") */ - prec = 0; - if (*f == '.') { - f++; - for (; Py_ISDIGIT(*f); f++) { - prec = (prec * 10) + (*f - '0'); - } - } - - while (*f && *f != '%' && !Py_ISALPHA(*f)) - f++; - - /* handle the long flag ('l'), but only for %ld and %lu. - others can be added when necessary. */ - longflag = 0; - if (*f == 'l' && (f[1] == 'd' || f[1] == 'u')) { - longflag = 1; - ++f; - } - - /* handle the size_t flag ('z'). */ - size_tflag = 0; - if (*f == 'z' && (f[1] == 'd' || f[1] == 'u')) { - size_tflag = 1; - ++f; - } - - /* subtract bytes preallocated for the format string - (ex: 2 for "%s") */ - writer.min_size -= (f - p + 1); - - switch (*f) { - case 'c': - { - int c = va_arg(vargs, int); - if (c < 0 || c > 255) { - PyErr_SetString(PyExc_OverflowError, - "PyBytes_FromFormatV(): %c format " - "expects an integer in range [0; 255]"); - goto error; - } - writer.min_size++; - *s++ = (unsigned char)c; - break; - } - - case 'd': - if (longflag) - sprintf(buffer, "%ld", va_arg(vargs, long)); - else if (size_tflag) - sprintf(buffer, "%" PY_FORMAT_SIZE_T "d", - va_arg(vargs, Py_ssize_t)); - else - sprintf(buffer, "%d", va_arg(vargs, int)); - assert(strlen(buffer) < sizeof(buffer)); - WRITE_BYTES(buffer); - break; - - case 'u': - if (longflag) - sprintf(buffer, "%lu", - va_arg(vargs, unsigned long)); - else if (size_tflag) - sprintf(buffer, "%" PY_FORMAT_SIZE_T "u", - va_arg(vargs, size_t)); - else - sprintf(buffer, "%u", - va_arg(vargs, unsigned int)); - assert(strlen(buffer) < sizeof(buffer)); - WRITE_BYTES(buffer); - break; - - case 'i': - sprintf(buffer, "%i", va_arg(vargs, int)); - assert(strlen(buffer) < sizeof(buffer)); - WRITE_BYTES(buffer); - break; - - case 'x': - sprintf(buffer, "%x", va_arg(vargs, int)); - assert(strlen(buffer) < sizeof(buffer)); - WRITE_BYTES(buffer); - break; - - case 's': - { - Py_ssize_t i; - - p = va_arg(vargs, const char*); - i = strlen(p); - if (prec > 0 && i > prec) - i = prec; - s = _PyBytesWriter_WriteBytes(&writer, s, p, i); - if (s == NULL) - goto error; - break; - } - - case 'p': - sprintf(buffer, "%p", va_arg(vargs, void*)); - assert(strlen(buffer) < sizeof(buffer)); - /* %p is ill-defined: ensure leading 0x. */ - if (buffer[1] == 'X') - buffer[1] = 'x'; - else if (buffer[1] != 'x') { - memmove(buffer+2, buffer, strlen(buffer)+1); - buffer[0] = '0'; - buffer[1] = 'x'; - } - WRITE_BYTES(buffer); - break; - - case '%': - writer.min_size++; - *s++ = '%'; - break; - - default: - if (*f == 0) { - /* fix min_size if we reached the end of the format string */ - writer.min_size++; - } - - /* invalid format string: copy unformatted string and exit */ - WRITE_BYTES(p); - return _PyBytesWriter_Finish(&writer, s); - } - } - -#undef WRITE_BYTES - - return _PyBytesWriter_Finish(&writer, s); - - error: - _PyBytesWriter_Dealloc(&writer); - return NULL; -} - -PyObject * -PyBytes_FromFormat(const char *format, ...) -{ - PyObject* ret; - va_list vargs; - -#ifdef HAVE_STDARG_PROTOTYPES - va_start(vargs, format); -#else - va_start(vargs); -#endif - ret = PyBytes_FromFormatV(format, vargs); - va_end(vargs); - return ret; -} - -/* Helpers for formatstring */ - -Py_LOCAL_INLINE(PyObject *) -getnextarg(PyObject *args, Py_ssize_t arglen, Py_ssize_t *p_argidx) -{ - Py_ssize_t argidx = *p_argidx; - if (argidx < arglen) { - (*p_argidx)++; - if (arglen < 0) - return args; - else - return PyTuple_GetItem(args, argidx); - } - PyErr_SetString(PyExc_TypeError, - "not enough arguments for format string"); - return NULL; -} - -/* Format codes - * F_LJUST '-' - * F_SIGN '+' - * F_BLANK ' ' - * F_ALT '#' - * F_ZERO '0' - */ -#define F_LJUST (1<<0) -#define F_SIGN (1<<1) -#define F_BLANK (1<<2) -#define F_ALT (1<<3) -#define F_ZERO (1<<4) - -/* Returns a new reference to a PyBytes object, or NULL on failure. */ - -static char* -formatfloat(PyObject *v, int flags, int prec, int type, - PyObject **p_result, _PyBytesWriter *writer, char *str) -{ - char *p; - PyObject *result; - double x; - size_t len; - - x = PyFloat_AsDouble(v); - if (x == -1.0 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, "float argument required, " - "not %.200s", Py_TYPE(v)->tp_name); - return NULL; - } - - if (prec < 0) - prec = 6; - - p = PyOS_double_to_string(x, type, prec, - (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); - - if (p == NULL) - return NULL; - - len = strlen(p); - if (writer != NULL) { - str = _PyBytesWriter_Prepare(writer, str, len); - if (str == NULL) - return NULL; - memcpy(str, p, len); - PyMem_Free(p); - str += len; - return str; - } - - result = PyBytes_FromStringAndSize(p, len); - PyMem_Free(p); - *p_result = result; - return str; -} - -static PyObject * -formatlong(PyObject *v, int flags, int prec, int type) -{ - PyObject *result, *iobj; - if (type == 'i') - type = 'd'; - if (PyLong_Check(v)) - return _PyUnicode_FormatLong(v, flags & F_ALT, prec, type); - if (PyNumber_Check(v)) { - /* make sure number is a type of integer for o, x, and X */ - if (type == 'o' || type == 'x' || type == 'X') - iobj = PyNumber_Index(v); - else - iobj = PyNumber_Long(v); - if (iobj == NULL) { - if (!PyErr_ExceptionMatches(PyExc_TypeError)) - return NULL; - } - else if (!PyLong_Check(iobj)) - Py_CLEAR(iobj); - if (iobj != NULL) { - result = _PyUnicode_FormatLong(iobj, flags & F_ALT, prec, type); - Py_DECREF(iobj); - return result; - } - } - PyErr_Format(PyExc_TypeError, - "%%%c format: %s is required, not %.200s", type, - (type == 'o' || type == 'x' || type == 'X') ? "an integer" - : "a number", - Py_TYPE(v)->tp_name); - return NULL; -} - -static int -byte_converter(PyObject *arg, char *p) -{ - if (PyBytes_Check(arg) && PyBytes_GET_SIZE(arg) == 1) { - *p = PyBytes_AS_STRING(arg)[0]; - return 1; - } - else if (PyByteArray_Check(arg) && PyByteArray_GET_SIZE(arg) == 1) { - *p = PyByteArray_AS_STRING(arg)[0]; - return 1; - } - else { - PyObject *iobj; - long ival; - int overflow; - /* make sure number is a type of integer */ - if (PyLong_Check(arg)) { - ival = PyLong_AsLongAndOverflow(arg, &overflow); - } - else { - iobj = PyNumber_Index(arg); - if (iobj == NULL) { - if (!PyErr_ExceptionMatches(PyExc_TypeError)) - return 0; - goto onError; - } - ival = PyLong_AsLongAndOverflow(iobj, &overflow); - Py_DECREF(iobj); - } - if (!overflow && ival == -1 && PyErr_Occurred()) - goto onError; - if (overflow || !(0 <= ival && ival <= 255)) { - PyErr_SetString(PyExc_OverflowError, - "%c arg not in range(256)"); - return 0; - } - *p = (char)ival; - return 1; - } - onError: - PyErr_SetString(PyExc_TypeError, - "%c requires an integer in range(256) or a single byte"); - return 0; -} - -static PyObject *_PyBytes_FromBuffer(PyObject *x); - -static PyObject * -format_obj(PyObject *v, const char **pbuf, Py_ssize_t *plen) -{ - PyObject *func, *result; - _Py_IDENTIFIER(__bytes__); - /* is it a bytes object? */ - if (PyBytes_Check(v)) { - *pbuf = PyBytes_AS_STRING(v); - *plen = PyBytes_GET_SIZE(v); - Py_INCREF(v); - return v; - } - if (PyByteArray_Check(v)) { - *pbuf = PyByteArray_AS_STRING(v); - *plen = PyByteArray_GET_SIZE(v); - Py_INCREF(v); - return v; - } - /* does it support __bytes__? */ - func = _PyObject_LookupSpecial(v, &PyId___bytes__); - if (func != NULL) { - result = _PyObject_CallNoArg(func); - Py_DECREF(func); - if (result == NULL) - return NULL; - if (!PyBytes_Check(result)) { - PyErr_Format(PyExc_TypeError, - "__bytes__ returned non-bytes (type %.200s)", - Py_TYPE(result)->tp_name); - Py_DECREF(result); - return NULL; - } - *pbuf = PyBytes_AS_STRING(result); - *plen = PyBytes_GET_SIZE(result); - return result; - } - /* does it support buffer protocol? */ - if (PyObject_CheckBuffer(v)) { - /* maybe we can avoid making a copy of the buffer object here? */ - result = _PyBytes_FromBuffer(v); - if (result == NULL) - return NULL; - *pbuf = PyBytes_AS_STRING(result); - *plen = PyBytes_GET_SIZE(result); - return result; - } - PyErr_Format(PyExc_TypeError, - "%%b requires a bytes-like object, " - "or an object that implements __bytes__, not '%.100s'", - Py_TYPE(v)->tp_name); - return NULL; -} - -/* fmt%(v1,v2,...) is roughly equivalent to sprintf(fmt, v1, v2, ...) */ - -PyObject * -_PyBytes_FormatEx(const char *format, Py_ssize_t format_len, - PyObject *args, int use_bytearray) -{ - const char *fmt; - char *res; - Py_ssize_t arglen, argidx; - Py_ssize_t fmtcnt; - int args_owned = 0; - PyObject *dict = NULL; - _PyBytesWriter writer; - - if (args == NULL) { - PyErr_BadInternalCall(); - return NULL; - } - fmt = format; - fmtcnt = format_len; - - _PyBytesWriter_Init(&writer); - writer.use_bytearray = use_bytearray; - - res = _PyBytesWriter_Alloc(&writer, fmtcnt); - if (res == NULL) - return NULL; - if (!use_bytearray) - writer.overallocate = 1; - - if (PyTuple_Check(args)) { - arglen = PyTuple_GET_SIZE(args); - argidx = 0; - } - else { - arglen = -1; - argidx = -2; - } - if (Py_TYPE(args)->tp_as_mapping && Py_TYPE(args)->tp_as_mapping->mp_subscript && - !PyTuple_Check(args) && !PyBytes_Check(args) && !PyUnicode_Check(args) && - !PyByteArray_Check(args)) { - dict = args; - } - - while (--fmtcnt >= 0) { - if (*fmt != '%') { - Py_ssize_t len; - char *pos; - - pos = (char *)memchr(fmt + 1, '%', fmtcnt); - if (pos != NULL) - len = pos - fmt; - else - len = fmtcnt + 1; - assert(len != 0); - - memcpy(res, fmt, len); - res += len; - fmt += len; - fmtcnt -= (len - 1); - } - else { - /* Got a format specifier */ - int flags = 0; - Py_ssize_t width = -1; - int prec = -1; - int c = '\0'; - int fill; - PyObject *v = NULL; - PyObject *temp = NULL; - const char *pbuf = NULL; - int sign; - Py_ssize_t len = 0; - char onechar; /* For byte_converter() */ - Py_ssize_t alloc; -#ifdef Py_DEBUG - char *before; -#endif - - fmt++; - if (*fmt == '%') { - *res++ = '%'; - fmt++; - fmtcnt--; - continue; - } - if (*fmt == '(') { - const char *keystart; - Py_ssize_t keylen; - PyObject *key; - int pcount = 1; - - if (dict == NULL) { - PyErr_SetString(PyExc_TypeError, - "format requires a mapping"); - goto error; - } - ++fmt; - --fmtcnt; - keystart = fmt; - /* Skip over balanced parentheses */ - while (pcount > 0 && --fmtcnt >= 0) { - if (*fmt == ')') - --pcount; - else if (*fmt == '(') - ++pcount; - fmt++; - } - keylen = fmt - keystart - 1; - if (fmtcnt < 0 || pcount > 0) { - PyErr_SetString(PyExc_ValueError, - "incomplete format key"); - goto error; - } - key = PyBytes_FromStringAndSize(keystart, - keylen); - if (key == NULL) - goto error; - if (args_owned) { - Py_DECREF(args); - args_owned = 0; - } - args = PyObject_GetItem(dict, key); - Py_DECREF(key); - if (args == NULL) { - goto error; - } - args_owned = 1; - arglen = -1; - argidx = -2; - } - - /* Parse flags. Example: "%+i" => flags=F_SIGN. */ - while (--fmtcnt >= 0) { - switch (c = *fmt++) { - case '-': flags |= F_LJUST; continue; - case '+': flags |= F_SIGN; continue; - case ' ': flags |= F_BLANK; continue; - case '#': flags |= F_ALT; continue; - case '0': flags |= F_ZERO; continue; - } - break; - } - - /* Parse width. Example: "%10s" => width=10 */ - if (c == '*') { - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - if (!PyLong_Check(v)) { - PyErr_SetString(PyExc_TypeError, - "* wants int"); - goto error; - } - width = PyLong_AsSsize_t(v); - if (width == -1 && PyErr_Occurred()) - goto error; - if (width < 0) { - flags |= F_LJUST; - width = -width; - } - if (--fmtcnt >= 0) - c = *fmt++; - } - else if (c >= 0 && isdigit(c)) { - width = c - '0'; - while (--fmtcnt >= 0) { - c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) - break; - if (width > (PY_SSIZE_T_MAX - ((int)c - '0')) / 10) { - PyErr_SetString( - PyExc_ValueError, - "width too big"); - goto error; - } - width = width*10 + (c - '0'); - } - } - - /* Parse precision. Example: "%.3f" => prec=3 */ - if (c == '.') { - prec = 0; - if (--fmtcnt >= 0) - c = *fmt++; - if (c == '*') { - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - if (!PyLong_Check(v)) { - PyErr_SetString( - PyExc_TypeError, - "* wants int"); - goto error; - } - prec = _PyLong_AsInt(v); - if (prec == -1 && PyErr_Occurred()) - goto error; - if (prec < 0) - prec = 0; - if (--fmtcnt >= 0) - c = *fmt++; - } - else if (c >= 0 && isdigit(c)) { - prec = c - '0'; - while (--fmtcnt >= 0) { - c = Py_CHARMASK(*fmt++); - if (!isdigit(c)) - break; - if (prec > (INT_MAX - ((int)c - '0')) / 10) { - PyErr_SetString( - PyExc_ValueError, - "prec too big"); - goto error; - } - prec = prec*10 + (c - '0'); - } - } - } /* prec */ - if (fmtcnt >= 0) { - if (c == 'h' || c == 'l' || c == 'L') { - if (--fmtcnt >= 0) - c = *fmt++; - } - } - if (fmtcnt < 0) { - PyErr_SetString(PyExc_ValueError, - "incomplete format"); - goto error; - } - v = getnextarg(args, arglen, &argidx); - if (v == NULL) - goto error; - - if (fmtcnt < 0) { - /* last writer: disable writer overallocation */ - writer.overallocate = 0; - } - - sign = 0; - fill = ' '; - switch (c) { - case 'r': - // %r is only for 2/3 code; 3 only code should use %a - case 'a': - temp = PyObject_ASCII(v); - if (temp == NULL) - goto error; - assert(PyUnicode_IS_ASCII(temp)); - pbuf = (const char *)PyUnicode_1BYTE_DATA(temp); - len = PyUnicode_GET_LENGTH(temp); - if (prec >= 0 && len > prec) - len = prec; - break; - - case 's': - // %s is only for 2/3 code; 3 only code should use %b - case 'b': - temp = format_obj(v, &pbuf, &len); - if (temp == NULL) - goto error; - if (prec >= 0 && len > prec) - len = prec; - break; - - case 'i': - case 'd': - case 'u': - case 'o': - case 'x': - case 'X': - if (PyLong_CheckExact(v) - && width == -1 && prec == -1 - && !(flags & (F_SIGN | F_BLANK)) - && c != 'X') - { - /* Fast path */ - int alternate = flags & F_ALT; - int base; - - switch(c) - { - default: - Py_UNREACHABLE(); - case 'd': - case 'i': - case 'u': - base = 10; - break; - case 'o': - base = 8; - break; - case 'x': - case 'X': - base = 16; - break; - } - - /* Fast path */ - writer.min_size -= 2; /* size preallocated for "%d" */ - res = _PyLong_FormatBytesWriter(&writer, res, - v, base, alternate); - if (res == NULL) - goto error; - continue; - } - - temp = formatlong(v, flags, prec, c); - if (!temp) - goto error; - assert(PyUnicode_IS_ASCII(temp)); - pbuf = (const char *)PyUnicode_1BYTE_DATA(temp); - len = PyUnicode_GET_LENGTH(temp); - sign = 1; - if (flags & F_ZERO) - fill = '0'; - break; - - case 'e': - case 'E': - case 'f': - case 'F': - case 'g': - case 'G': - if (width == -1 && prec == -1 - && !(flags & (F_SIGN | F_BLANK))) - { - /* Fast path */ - writer.min_size -= 2; /* size preallocated for "%f" */ - res = formatfloat(v, flags, prec, c, NULL, &writer, res); - if (res == NULL) - goto error; - continue; - } - - if (!formatfloat(v, flags, prec, c, &temp, NULL, res)) - goto error; - pbuf = PyBytes_AS_STRING(temp); - len = PyBytes_GET_SIZE(temp); - sign = 1; - if (flags & F_ZERO) - fill = '0'; - break; - - case 'c': - pbuf = &onechar; - len = byte_converter(v, &onechar); - if (!len) - goto error; - if (width == -1) { - /* Fast path */ - *res++ = onechar; - continue; - } - break; - - default: - PyErr_Format(PyExc_ValueError, - "unsupported format character '%c' (0x%x) " - "at index %zd", - c, c, - (Py_ssize_t)(fmt - 1 - format)); - goto error; - } - - if (sign) { - if (*pbuf == '-' || *pbuf == '+') { - sign = *pbuf++; - len--; - } - else if (flags & F_SIGN) - sign = '+'; - else if (flags & F_BLANK) - sign = ' '; - else - sign = 0; - } - if (width < len) - width = len; - - alloc = width; - if (sign != 0 && len == width) - alloc++; - /* 2: size preallocated for %s */ - if (alloc > 2) { - res = _PyBytesWriter_Prepare(&writer, res, alloc - 2); - if (res == NULL) - goto error; - } -#ifdef Py_DEBUG - before = res; -#endif - - /* Write the sign if needed */ - if (sign) { - if (fill != ' ') - *res++ = sign; - if (width > len) - width--; - } - - /* Write the numeric prefix for "x", "X" and "o" formats - if the alternate form is used. - For example, write "0x" for the "%#x" format. */ - if ((flags & F_ALT) && (c == 'o' || c == 'x' || c == 'X')) { - assert(pbuf[0] == '0'); - assert(pbuf[1] == c); - if (fill != ' ') { - *res++ = *pbuf++; - *res++ = *pbuf++; - } - width -= 2; - if (width < 0) - width = 0; - len -= 2; - } - - /* Pad left with the fill character if needed */ - if (width > len && !(flags & F_LJUST)) { - memset(res, fill, width - len); - res += (width - len); - width = len; - } - - /* If padding with spaces: write sign if needed and/or numeric - prefix if the alternate form is used */ - if (fill == ' ') { - if (sign) - *res++ = sign; - if ((flags & F_ALT) && (c == 'o' || c == 'x' || c == 'X')) { - assert(pbuf[0] == '0'); - assert(pbuf[1] == c); - *res++ = *pbuf++; - *res++ = *pbuf++; - } - } - - /* Copy bytes */ - memcpy(res, pbuf, len); - res += len; - - /* Pad right with the fill character if needed */ - if (width > len) { - memset(res, ' ', width - len); - res += (width - len); - } - - if (dict && (argidx < arglen)) { - PyErr_SetString(PyExc_TypeError, - "not all arguments converted during bytes formatting"); - Py_XDECREF(temp); - goto error; - } - Py_XDECREF(temp); - -#ifdef Py_DEBUG - /* check that we computed the exact size for this write */ - assert((res - before) == alloc); -#endif - } /* '%' */ - - /* If overallocation was disabled, ensure that it was the last - write. Otherwise, we missed an optimization */ - assert(writer.overallocate || fmtcnt < 0 || use_bytearray); - } /* until end */ - - if (argidx < arglen && !dict) { - PyErr_SetString(PyExc_TypeError, - "not all arguments converted during bytes formatting"); - goto error; - } - - if (args_owned) { - Py_DECREF(args); - } - return _PyBytesWriter_Finish(&writer, res); - - error: - _PyBytesWriter_Dealloc(&writer); - if (args_owned) { - Py_DECREF(args); - } - return NULL; -} - -/* =-= */ - -static void -bytes_dealloc(PyObject *op) -{ - Py_TYPE(op)->tp_free(op); -} - -/* Unescape a backslash-escaped string. If unicode is non-zero, - the string is a u-literal. If recode_encoding is non-zero, - the string is UTF-8 encoded and should be re-encoded in the - specified encoding. */ - -static char * -_PyBytes_DecodeEscapeRecode(const char **s, const char *end, - const char *errors, const char *recode_encoding, - _PyBytesWriter *writer, char *p) -{ - PyObject *u, *w; - const char* t; - - t = *s; - /* Decode non-ASCII bytes as UTF-8. */ - while (t < end && (*t & 0x80)) - t++; - u = PyUnicode_DecodeUTF8(*s, t - *s, errors); - if (u == NULL) - return NULL; - - /* Recode them in target encoding. */ - w = PyUnicode_AsEncodedString(u, recode_encoding, errors); - Py_DECREF(u); - if (w == NULL) - return NULL; - assert(PyBytes_Check(w)); - - /* Append bytes to output buffer. */ - writer->min_size--; /* subtract 1 preallocated byte */ - p = _PyBytesWriter_WriteBytes(writer, p, - PyBytes_AS_STRING(w), - PyBytes_GET_SIZE(w)); - Py_DECREF(w); - if (p == NULL) - return NULL; - - *s = t; - return p; -} - -PyObject *_PyBytes_DecodeEscape(const char *s, - Py_ssize_t len, - const char *errors, - Py_ssize_t unicode, - const char *recode_encoding, - const char **first_invalid_escape) -{ - int c; - char *p; - const char *end; - _PyBytesWriter writer; - - _PyBytesWriter_Init(&writer); - - p = _PyBytesWriter_Alloc(&writer, len); - if (p == NULL) - return NULL; - writer.overallocate = 1; - - *first_invalid_escape = NULL; - - end = s + len; - while (s < end) { - if (*s != '\\') { - non_esc: - if (!(recode_encoding && (*s & 0x80))) { - *p++ = *s++; - } - else { - /* non-ASCII character and need to recode */ - p = _PyBytes_DecodeEscapeRecode(&s, end, - errors, recode_encoding, - &writer, p); - if (p == NULL) - goto failed; - } - continue; - } - - s++; - if (s == end) { - PyErr_SetString(PyExc_ValueError, - "Trailing \\ in string"); - goto failed; - } - - switch (*s++) { - /* XXX This assumes ASCII! */ - case '\n': break; - case '\\': *p++ = '\\'; break; - case '\'': *p++ = '\''; break; - case '\"': *p++ = '\"'; break; - case 'b': *p++ = '\b'; break; - case 'f': *p++ = '\014'; break; /* FF */ - case 't': *p++ = '\t'; break; - case 'n': *p++ = '\n'; break; - case 'r': *p++ = '\r'; break; - case 'v': *p++ = '\013'; break; /* VT */ - case 'a': *p++ = '\007'; break; /* BEL, not classic C */ - case '0': case '1': case '2': case '3': - case '4': case '5': case '6': case '7': - c = s[-1] - '0'; - if (s < end && '0' <= *s && *s <= '7') { - c = (c<<3) + *s++ - '0'; - if (s < end && '0' <= *s && *s <= '7') - c = (c<<3) + *s++ - '0'; - } - *p++ = c; - break; - case 'x': - if (s+1 < end) { - int digit1, digit2; - digit1 = _PyLong_DigitValue[Py_CHARMASK(s[0])]; - digit2 = _PyLong_DigitValue[Py_CHARMASK(s[1])]; - if (digit1 < 16 && digit2 < 16) { - *p++ = (unsigned char)((digit1 << 4) + digit2); - s += 2; - break; - } - } - /* invalid hexadecimal digits */ - - if (!errors || strcmp(errors, "strict") == 0) { - PyErr_Format(PyExc_ValueError, - "invalid \\x escape at position %d", - s - 2 - (end - len)); - goto failed; - } - if (strcmp(errors, "replace") == 0) { - *p++ = '?'; - } else if (strcmp(errors, "ignore") == 0) - /* do nothing */; - else { - PyErr_Format(PyExc_ValueError, - "decoding error; unknown " - "error handling code: %.400s", - errors); - goto failed; - } - /* skip \x */ - if (s < end && Py_ISXDIGIT(s[0])) - s++; /* and a hexdigit */ - break; - - default: - if (*first_invalid_escape == NULL) { - *first_invalid_escape = s-1; /* Back up one char, since we've - already incremented s. */ - } - *p++ = '\\'; - s--; - goto non_esc; /* an arbitrary number of unescaped - UTF-8 bytes may follow. */ - } - } - - return _PyBytesWriter_Finish(&writer, p); - - failed: - _PyBytesWriter_Dealloc(&writer); - return NULL; -} - -PyObject *PyBytes_DecodeEscape(const char *s, - Py_ssize_t len, - const char *errors, - Py_ssize_t unicode, - const char *recode_encoding) -{ - const char* first_invalid_escape; - PyObject *result = _PyBytes_DecodeEscape(s, len, errors, unicode, - recode_encoding, - &first_invalid_escape); - if (result == NULL) - return NULL; - if (first_invalid_escape != NULL) { - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "invalid escape sequence '\\%c'", - (unsigned char)*first_invalid_escape) < 0) { - Py_DECREF(result); - return NULL; - } - } - return result; - -} -/* -------------------------------------------------------------------- */ -/* object api */ - -Py_ssize_t -PyBytes_Size(PyObject *op) -{ - if (!PyBytes_Check(op)) { - PyErr_Format(PyExc_TypeError, - "expected bytes, %.200s found", Py_TYPE(op)->tp_name); - return -1; - } - return Py_SIZE(op); -} - -char * -PyBytes_AsString(PyObject *op) -{ - if (!PyBytes_Check(op)) { - PyErr_Format(PyExc_TypeError, - "expected bytes, %.200s found", Py_TYPE(op)->tp_name); - return NULL; - } - return ((PyBytesObject *)op)->ob_sval; -} - -int -PyBytes_AsStringAndSize(PyObject *obj, - char **s, - Py_ssize_t *len) -{ - if (s == NULL) { - PyErr_BadInternalCall(); - return -1; - } - - if (!PyBytes_Check(obj)) { - PyErr_Format(PyExc_TypeError, - "expected bytes, %.200s found", Py_TYPE(obj)->tp_name); - return -1; - } - - *s = PyBytes_AS_STRING(obj); - if (len != NULL) - *len = PyBytes_GET_SIZE(obj); - else if (strlen(*s) != (size_t)PyBytes_GET_SIZE(obj)) { - PyErr_SetString(PyExc_ValueError, - "embedded null byte"); - return -1; - } - return 0; -} - -/* -------------------------------------------------------------------- */ -/* Methods */ - -#include "stringlib/stringdefs.h" - -#include "stringlib/fastsearch.h" -#include "stringlib/count.h" -#include "stringlib/find.h" -#include "stringlib/join.h" -#include "stringlib/partition.h" -#include "stringlib/split.h" -#include "stringlib/ctype.h" - -#include "stringlib/transmogrify.h" - -PyObject * -PyBytes_Repr(PyObject *obj, int smartquotes) -{ - PyBytesObject* op = (PyBytesObject*) obj; - Py_ssize_t i, length = Py_SIZE(op); - Py_ssize_t newsize, squotes, dquotes; - PyObject *v; - unsigned char quote, *s, *p; - - /* Compute size of output string */ - squotes = dquotes = 0; - newsize = 3; /* b'' */ - s = (unsigned char*)op->ob_sval; - for (i = 0; i < length; i++) { - Py_ssize_t incr = 1; - switch(s[i]) { - case '\'': squotes++; break; - case '"': dquotes++; break; - case '\\': case '\t': case '\n': case '\r': - incr = 2; break; /* \C */ - default: - if (s[i] < ' ' || s[i] >= 0x7f) - incr = 4; /* \xHH */ - } - if (newsize > PY_SSIZE_T_MAX - incr) - goto overflow; - newsize += incr; - } - quote = '\''; - if (smartquotes && squotes && !dquotes) - quote = '"'; - if (squotes && quote == '\'') { - if (newsize > PY_SSIZE_T_MAX - squotes) - goto overflow; - newsize += squotes; - } - - v = PyUnicode_New(newsize, 127); - if (v == NULL) { - return NULL; - } - p = PyUnicode_1BYTE_DATA(v); - - *p++ = 'b', *p++ = quote; - for (i = 0; i < length; i++) { - unsigned char c = op->ob_sval[i]; - if (c == quote || c == '\\') - *p++ = '\\', *p++ = c; - else if (c == '\t') - *p++ = '\\', *p++ = 't'; - else if (c == '\n') - *p++ = '\\', *p++ = 'n'; - else if (c == '\r') - *p++ = '\\', *p++ = 'r'; - else if (c < ' ' || c >= 0x7f) { - *p++ = '\\'; - *p++ = 'x'; - *p++ = Py_hexdigits[(c & 0xf0) >> 4]; - *p++ = Py_hexdigits[c & 0xf]; - } - else - *p++ = c; - } - *p++ = quote; - assert(_PyUnicode_CheckConsistency(v, 1)); - return v; - - overflow: - PyErr_SetString(PyExc_OverflowError, - "bytes object is too large to make repr"); - return NULL; -} - -static PyObject * -bytes_repr(PyObject *op) -{ - return PyBytes_Repr(op, 1); -} - -static PyObject * -bytes_str(PyObject *op) -{ - if (Py_BytesWarningFlag) { - if (PyErr_WarnEx(PyExc_BytesWarning, - "str() on a bytes instance", 1)) - return NULL; - } - return bytes_repr(op); -} - -static Py_ssize_t -bytes_length(PyBytesObject *a) -{ - return Py_SIZE(a); -} - -/* This is also used by PyBytes_Concat() */ -static PyObject * -bytes_concat(PyObject *a, PyObject *b) -{ - Py_buffer va, vb; - PyObject *result = NULL; - - va.len = -1; - vb.len = -1; - if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 || - PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) { - PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s", - Py_TYPE(b)->tp_name, Py_TYPE(a)->tp_name); - goto done; - } - - /* Optimize end cases */ - if (va.len == 0 && PyBytes_CheckExact(b)) { - result = b; - Py_INCREF(result); - goto done; - } - if (vb.len == 0 && PyBytes_CheckExact(a)) { - result = a; - Py_INCREF(result); - goto done; - } - - if (va.len > PY_SSIZE_T_MAX - vb.len) { - PyErr_NoMemory(); - goto done; - } - - result = PyBytes_FromStringAndSize(NULL, va.len + vb.len); - if (result != NULL) { - memcpy(PyBytes_AS_STRING(result), va.buf, va.len); - memcpy(PyBytes_AS_STRING(result) + va.len, vb.buf, vb.len); - } - - done: - if (va.len != -1) - PyBuffer_Release(&va); - if (vb.len != -1) - PyBuffer_Release(&vb); - return result; -} - -static PyObject * -bytes_repeat(PyBytesObject *a, Py_ssize_t n) -{ - Py_ssize_t i; - Py_ssize_t j; - Py_ssize_t size; - PyBytesObject *op; - size_t nbytes; - if (n < 0) - n = 0; - /* watch out for overflows: the size can overflow int, - * and the # of bytes needed can overflow size_t - */ - if (n > 0 && Py_SIZE(a) > PY_SSIZE_T_MAX / n) { - PyErr_SetString(PyExc_OverflowError, - "repeated bytes are too long"); - return NULL; - } - size = Py_SIZE(a) * n; - if (size == Py_SIZE(a) && PyBytes_CheckExact(a)) { - Py_INCREF(a); - return (PyObject *)a; - } - nbytes = (size_t)size; - if (nbytes + PyBytesObject_SIZE <= nbytes) { - PyErr_SetString(PyExc_OverflowError, - "repeated bytes are too long"); - return NULL; - } - op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + nbytes); - if (op == NULL) - return PyErr_NoMemory(); - (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); - op->ob_shash = -1; - op->ob_sval[size] = '\0'; - if (Py_SIZE(a) == 1 && n > 0) { - memset(op->ob_sval, a->ob_sval[0] , n); - return (PyObject *) op; - } - i = 0; - if (i < size) { - memcpy(op->ob_sval, a->ob_sval, Py_SIZE(a)); - i = Py_SIZE(a); - } - while (i < size) { - j = (i <= size-i) ? i : size-i; - memcpy(op->ob_sval+i, op->ob_sval, j); - i += j; - } - return (PyObject *) op; -} - -static int -bytes_contains(PyObject *self, PyObject *arg) -{ - return _Py_bytes_contains(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), arg); -} - -static PyObject * -bytes_item(PyBytesObject *a, Py_ssize_t i) -{ - if (i < 0 || i >= Py_SIZE(a)) { - PyErr_SetString(PyExc_IndexError, "index out of range"); - return NULL; - } - return PyLong_FromLong((unsigned char)a->ob_sval[i]); -} - -static int -bytes_compare_eq(PyBytesObject *a, PyBytesObject *b) -{ - int cmp; - Py_ssize_t len; - - len = Py_SIZE(a); - if (Py_SIZE(b) != len) - return 0; - - if (a->ob_sval[0] != b->ob_sval[0]) - return 0; - - cmp = memcmp(a->ob_sval, b->ob_sval, len); - return (cmp == 0); -} - -static PyObject* -bytes_richcompare(PyBytesObject *a, PyBytesObject *b, int op) -{ - int c; - Py_ssize_t len_a, len_b; - Py_ssize_t min_len; - PyObject *result; - int rc; - - /* Make sure both arguments are strings. */ - if (!(PyBytes_Check(a) && PyBytes_Check(b))) { - if (Py_BytesWarningFlag && (op == Py_EQ || op == Py_NE)) { - rc = PyObject_IsInstance((PyObject*)a, - (PyObject*)&PyUnicode_Type); - if (!rc) - rc = PyObject_IsInstance((PyObject*)b, - (PyObject*)&PyUnicode_Type); - if (rc < 0) - return NULL; - if (rc) { - if (PyErr_WarnEx(PyExc_BytesWarning, - "Comparison between bytes and string", 1)) - return NULL; - } - else { - rc = PyObject_IsInstance((PyObject*)a, - (PyObject*)&PyLong_Type); - if (!rc) - rc = PyObject_IsInstance((PyObject*)b, - (PyObject*)&PyLong_Type); - if (rc < 0) - return NULL; - if (rc) { - if (PyErr_WarnEx(PyExc_BytesWarning, - "Comparison between bytes and int", 1)) - return NULL; - } - } - } - result = Py_NotImplemented; - } - else if (a == b) { - switch (op) { - case Py_EQ: - case Py_LE: - case Py_GE: - /* a string is equal to itself */ - result = Py_True; - break; - case Py_NE: - case Py_LT: - case Py_GT: - result = Py_False; - break; - default: - PyErr_BadArgument(); - return NULL; - } - } - else if (op == Py_EQ || op == Py_NE) { - int eq = bytes_compare_eq(a, b); - eq ^= (op == Py_NE); - result = eq ? Py_True : Py_False; - } - else { - len_a = Py_SIZE(a); - len_b = Py_SIZE(b); - min_len = Py_MIN(len_a, len_b); - if (min_len > 0) { - c = Py_CHARMASK(*a->ob_sval) - Py_CHARMASK(*b->ob_sval); - if (c == 0) - c = memcmp(a->ob_sval, b->ob_sval, min_len); - } - else - c = 0; - if (c == 0) - c = (len_a < len_b) ? -1 : (len_a > len_b) ? 1 : 0; - switch (op) { - case Py_LT: c = c < 0; break; - case Py_LE: c = c <= 0; break; - case Py_GT: c = c > 0; break; - case Py_GE: c = c >= 0; break; - default: - PyErr_BadArgument(); - return NULL; - } - result = c ? Py_True : Py_False; - } - - Py_INCREF(result); - return result; -} - -static Py_hash_t -bytes_hash(PyBytesObject *a) -{ - if (a->ob_shash == -1) { - /* Can't fail */ - a->ob_shash = _Py_HashBytes(a->ob_sval, Py_SIZE(a)); - } - return a->ob_shash; -} - -static PyObject* -bytes_subscript(PyBytesObject* self, PyObject* item) -{ - if (PyIndex_Check(item)) { - Py_ssize_t i = PyNumber_AsSsize_t(item, PyExc_IndexError); - if (i == -1 && PyErr_Occurred()) - return NULL; - if (i < 0) - i += PyBytes_GET_SIZE(self); - if (i < 0 || i >= PyBytes_GET_SIZE(self)) { - PyErr_SetString(PyExc_IndexError, - "index out of range"); - return NULL; - } - return PyLong_FromLong((unsigned char)self->ob_sval[i]); - } - else if (PySlice_Check(item)) { - Py_ssize_t start, stop, step, slicelength, cur, i; - char* source_buf; - char* result_buf; - PyObject* result; - - if (PySlice_Unpack(item, &start, &stop, &step) < 0) { - return NULL; - } - slicelength = PySlice_AdjustIndices(PyBytes_GET_SIZE(self), &start, - &stop, step); - - if (slicelength <= 0) { - return PyBytes_FromStringAndSize("", 0); - } - else if (start == 0 && step == 1 && - slicelength == PyBytes_GET_SIZE(self) && - PyBytes_CheckExact(self)) { - Py_INCREF(self); - return (PyObject *)self; - } - else if (step == 1) { - return PyBytes_FromStringAndSize( - PyBytes_AS_STRING(self) + start, - slicelength); - } - else { - source_buf = PyBytes_AS_STRING(self); - result = PyBytes_FromStringAndSize(NULL, slicelength); - if (result == NULL) - return NULL; - - result_buf = PyBytes_AS_STRING(result); - for (cur = start, i = 0; i < slicelength; - cur += step, i++) { - result_buf[i] = source_buf[cur]; - } - - return result; - } - } - else { - PyErr_Format(PyExc_TypeError, - "byte indices must be integers or slices, not %.200s", - Py_TYPE(item)->tp_name); - return NULL; - } -} - -static int -bytes_buffer_getbuffer(PyBytesObject *self, Py_buffer *view, int flags) -{ - return PyBuffer_FillInfo(view, (PyObject*)self, (void *)self->ob_sval, Py_SIZE(self), - 1, flags); -} - -static PySequenceMethods bytes_as_sequence = { - (lenfunc)bytes_length, /*sq_length*/ - (binaryfunc)bytes_concat, /*sq_concat*/ - (ssizeargfunc)bytes_repeat, /*sq_repeat*/ - (ssizeargfunc)bytes_item, /*sq_item*/ - 0, /*sq_slice*/ - 0, /*sq_ass_item*/ - 0, /*sq_ass_slice*/ - (objobjproc)bytes_contains /*sq_contains*/ -}; - -static PyMappingMethods bytes_as_mapping = { - (lenfunc)bytes_length, - (binaryfunc)bytes_subscript, - 0, -}; - -static PyBufferProcs bytes_as_buffer = { - (getbufferproc)bytes_buffer_getbuffer, - NULL, -}; - - -#define LEFTSTRIP 0 -#define RIGHTSTRIP 1 -#define BOTHSTRIP 2 - -/*[clinic input] -bytes.split - - sep: object = None - The delimiter according which to split the bytes. - None (the default value) means split on ASCII whitespace characters - (space, tab, return, newline, formfeed, vertical tab). - maxsplit: Py_ssize_t = -1 - Maximum number of splits to do. - -1 (the default value) means no limit. - -Return a list of the sections in the bytes, using sep as the delimiter. -[clinic start generated code]*/ - -static PyObject * -bytes_split_impl(PyBytesObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=52126b5844c1d8ef input=8b809b39074abbfa]*/ -{ - Py_ssize_t len = PyBytes_GET_SIZE(self), n; - const char *s = PyBytes_AS_STRING(self), *sub; - Py_buffer vsub; - PyObject *list; - - if (maxsplit < 0) - maxsplit = PY_SSIZE_T_MAX; - if (sep == Py_None) - return stringlib_split_whitespace((PyObject*) self, s, len, maxsplit); - if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0) - return NULL; - sub = vsub.buf; - n = vsub.len; - - list = stringlib_split((PyObject*) self, s, len, sub, n, maxsplit); - PyBuffer_Release(&vsub); - return list; -} - -/*[clinic input] -bytes.partition - - sep: Py_buffer - / - -Partition the bytes into three parts using the given separator. - -This will search for the separator sep in the bytes. If the separator is found, -returns a 3-tuple containing the part before the separator, the separator -itself, and the part after it. - -If the separator is not found, returns a 3-tuple containing the original bytes -object and two empty bytes objects. -[clinic start generated code]*/ - -static PyObject * -bytes_partition_impl(PyBytesObject *self, Py_buffer *sep) -/*[clinic end generated code: output=f532b392a17ff695 input=61cca95519406099]*/ -{ - return stringlib_partition( - (PyObject*) self, - PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), - sep->obj, (const char *)sep->buf, sep->len - ); -} - -/*[clinic input] -bytes.rpartition - - sep: Py_buffer - / - -Partition the bytes into three parts using the given separator. - -This will search for the separator sep in the bytes, starting at the end. If -the separator is found, returns a 3-tuple containing the part before the -separator, the separator itself, and the part after it. - -If the separator is not found, returns a 3-tuple containing two empty bytes -objects and the original bytes object. -[clinic start generated code]*/ - -static PyObject * -bytes_rpartition_impl(PyBytesObject *self, Py_buffer *sep) -/*[clinic end generated code: output=191b114cbb028e50 input=d78db010c8cfdbe1]*/ -{ - return stringlib_rpartition( - (PyObject*) self, - PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), - sep->obj, (const char *)sep->buf, sep->len - ); -} - -/*[clinic input] -bytes.rsplit = bytes.split - -Return a list of the sections in the bytes, using sep as the delimiter. - -Splitting is done starting at the end of the bytes and working to the front. -[clinic start generated code]*/ - -static PyObject * -bytes_rsplit_impl(PyBytesObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=ba698d9ea01e1c8f input=0f86c9f28f7d7b7b]*/ -{ - Py_ssize_t len = PyBytes_GET_SIZE(self), n; - const char *s = PyBytes_AS_STRING(self), *sub; - Py_buffer vsub; - PyObject *list; - - if (maxsplit < 0) - maxsplit = PY_SSIZE_T_MAX; - if (sep == Py_None) - return stringlib_rsplit_whitespace((PyObject*) self, s, len, maxsplit); - if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0) - return NULL; - sub = vsub.buf; - n = vsub.len; - - list = stringlib_rsplit((PyObject*) self, s, len, sub, n, maxsplit); - PyBuffer_Release(&vsub); - return list; -} - - -/*[clinic input] -bytes.join - - iterable_of_bytes: object - / - -Concatenate any number of bytes objects. - -The bytes whose method is called is inserted in between each pair. - -The result is returned as a new bytes object. - -Example: b'.'.join([b'ab', b'pq', b'rs']) -> b'ab.pq.rs'. -[clinic start generated code]*/ - -static PyObject * -bytes_join(PyBytesObject *self, PyObject *iterable_of_bytes) -/*[clinic end generated code: output=a046f379f626f6f8 input=7fe377b95bd549d2]*/ -{ - return stringlib_bytes_join((PyObject*)self, iterable_of_bytes); -} - -PyObject * -_PyBytes_Join(PyObject *sep, PyObject *x) -{ - assert(sep != NULL && PyBytes_Check(sep)); - assert(x != NULL); - return bytes_join((PyBytesObject*)sep, x); -} - -static PyObject * -bytes_find(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_find(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - -static PyObject * -bytes_index(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_index(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - - -static PyObject * -bytes_rfind(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_rfind(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - - -static PyObject * -bytes_rindex(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_rindex(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - - -Py_LOCAL_INLINE(PyObject *) -do_xstrip(PyBytesObject *self, int striptype, PyObject *sepobj) -{ - Py_buffer vsep; - char *s = PyBytes_AS_STRING(self); - Py_ssize_t len = PyBytes_GET_SIZE(self); - char *sep; - Py_ssize_t seplen; - Py_ssize_t i, j; - - if (PyObject_GetBuffer(sepobj, &vsep, PyBUF_SIMPLE) != 0) - return NULL; - sep = vsep.buf; - seplen = vsep.len; - - i = 0; - if (striptype != RIGHTSTRIP) { - while (i < len && memchr(sep, Py_CHARMASK(s[i]), seplen)) { - i++; - } - } - - j = len; - if (striptype != LEFTSTRIP) { - do { - j--; - } while (j >= i && memchr(sep, Py_CHARMASK(s[j]), seplen)); - j++; - } - - PyBuffer_Release(&vsep); - - if (i == 0 && j == len && PyBytes_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*)self; - } - else - return PyBytes_FromStringAndSize(s+i, j-i); -} - - -Py_LOCAL_INLINE(PyObject *) -do_strip(PyBytesObject *self, int striptype) -{ - char *s = PyBytes_AS_STRING(self); - Py_ssize_t len = PyBytes_GET_SIZE(self), i, j; - - i = 0; - if (striptype != RIGHTSTRIP) { - while (i < len && Py_ISSPACE(s[i])) { - i++; - } - } - - j = len; - if (striptype != LEFTSTRIP) { - do { - j--; - } while (j >= i && Py_ISSPACE(s[j])); - j++; - } - - if (i == 0 && j == len && PyBytes_CheckExact(self)) { - Py_INCREF(self); - return (PyObject*)self; - } - else - return PyBytes_FromStringAndSize(s+i, j-i); -} - - -Py_LOCAL_INLINE(PyObject *) -do_argstrip(PyBytesObject *self, int striptype, PyObject *bytes) -{ - if (bytes != NULL && bytes != Py_None) { - return do_xstrip(self, striptype, bytes); - } - return do_strip(self, striptype); -} - -/*[clinic input] -bytes.strip - - bytes: object = None - / - -Strip leading and trailing bytes contained in the argument. - -If the argument is omitted or None, strip leading and trailing ASCII whitespace. -[clinic start generated code]*/ - -static PyObject * -bytes_strip_impl(PyBytesObject *self, PyObject *bytes) -/*[clinic end generated code: output=c7c228d3bd104a1b input=8a354640e4e0b3ef]*/ -{ - return do_argstrip(self, BOTHSTRIP, bytes); -} - -/*[clinic input] -bytes.lstrip - - bytes: object = None - / - -Strip leading bytes contained in the argument. - -If the argument is omitted or None, strip leading ASCII whitespace. -[clinic start generated code]*/ - -static PyObject * -bytes_lstrip_impl(PyBytesObject *self, PyObject *bytes) -/*[clinic end generated code: output=28602e586f524e82 input=9baff4398c3f6857]*/ -{ - return do_argstrip(self, LEFTSTRIP, bytes); -} - -/*[clinic input] -bytes.rstrip - - bytes: object = None - / - -Strip trailing bytes contained in the argument. - -If the argument is omitted or None, strip trailing ASCII whitespace. -[clinic start generated code]*/ - -static PyObject * -bytes_rstrip_impl(PyBytesObject *self, PyObject *bytes) -/*[clinic end generated code: output=547e3815c95447da input=b78af445c727e32b]*/ -{ - return do_argstrip(self, RIGHTSTRIP, bytes); -} - - -static PyObject * -bytes_count(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_count(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - - -/*[clinic input] -bytes.translate - - table: object - Translation table, which must be a bytes object of length 256. - / - delete as deletechars: object(c_default="NULL") = b'' - -Return a copy with each character mapped by the given translation table. - -All characters occurring in the optional argument delete are removed. -The remaining characters are mapped through the given translation table. -[clinic start generated code]*/ - -static PyObject * -bytes_translate_impl(PyBytesObject *self, PyObject *table, - PyObject *deletechars) -/*[clinic end generated code: output=43be3437f1956211 input=0ecdf159f654233c]*/ -{ - char *input, *output; - Py_buffer table_view = {NULL, NULL}; - Py_buffer del_table_view = {NULL, NULL}; - const char *table_chars; - Py_ssize_t i, c, changed = 0; - PyObject *input_obj = (PyObject*)self; - const char *output_start, *del_table_chars=NULL; - Py_ssize_t inlen, tablen, dellen = 0; - PyObject *result; - int trans_table[256]; - - if (PyBytes_Check(table)) { - table_chars = PyBytes_AS_STRING(table); - tablen = PyBytes_GET_SIZE(table); - } - else if (table == Py_None) { - table_chars = NULL; - tablen = 256; - } - else { - if (PyObject_GetBuffer(table, &table_view, PyBUF_SIMPLE) != 0) - return NULL; - table_chars = table_view.buf; - tablen = table_view.len; - } - - if (tablen != 256) { - PyErr_SetString(PyExc_ValueError, - "translation table must be 256 characters long"); - PyBuffer_Release(&table_view); - return NULL; - } - - if (deletechars != NULL) { - if (PyBytes_Check(deletechars)) { - del_table_chars = PyBytes_AS_STRING(deletechars); - dellen = PyBytes_GET_SIZE(deletechars); - } - else { - if (PyObject_GetBuffer(deletechars, &del_table_view, PyBUF_SIMPLE) != 0) { - PyBuffer_Release(&table_view); - return NULL; - } - del_table_chars = del_table_view.buf; - dellen = del_table_view.len; - } - } - else { - del_table_chars = NULL; - dellen = 0; - } - - inlen = PyBytes_GET_SIZE(input_obj); - result = PyBytes_FromStringAndSize((char *)NULL, inlen); - if (result == NULL) { - PyBuffer_Release(&del_table_view); - PyBuffer_Release(&table_view); - return NULL; - } - output_start = output = PyBytes_AS_STRING(result); - input = PyBytes_AS_STRING(input_obj); - - if (dellen == 0 && table_chars != NULL) { - /* If no deletions are required, use faster code */ - for (i = inlen; --i >= 0; ) { - c = Py_CHARMASK(*input++); - if (Py_CHARMASK((*output++ = table_chars[c])) != c) - changed = 1; - } - if (!changed && PyBytes_CheckExact(input_obj)) { - Py_INCREF(input_obj); - Py_DECREF(result); - result = input_obj; - } - PyBuffer_Release(&del_table_view); - PyBuffer_Release(&table_view); - return result; - } - - if (table_chars == NULL) { - for (i = 0; i < 256; i++) - trans_table[i] = Py_CHARMASK(i); - } else { - for (i = 0; i < 256; i++) - trans_table[i] = Py_CHARMASK(table_chars[i]); - } - PyBuffer_Release(&table_view); - - for (i = 0; i < dellen; i++) - trans_table[(int) Py_CHARMASK(del_table_chars[i])] = -1; - PyBuffer_Release(&del_table_view); - - for (i = inlen; --i >= 0; ) { - c = Py_CHARMASK(*input++); - if (trans_table[c] != -1) - if (Py_CHARMASK(*output++ = (char)trans_table[c]) == c) - continue; - changed = 1; - } - if (!changed && PyBytes_CheckExact(input_obj)) { - Py_DECREF(result); - Py_INCREF(input_obj); - return input_obj; - } - /* Fix the size of the resulting string */ - if (inlen > 0) - _PyBytes_Resize(&result, output - output_start); - return result; -} - - -/*[clinic input] - -@staticmethod -bytes.maketrans - - frm: Py_buffer - to: Py_buffer - / - -Return a translation table useable for the bytes or bytearray translate method. - -The returned table will be one where each byte in frm is mapped to the byte at -the same position in to. - -The bytes objects frm and to must be of the same length. -[clinic start generated code]*/ - -static PyObject * -bytes_maketrans_impl(Py_buffer *frm, Py_buffer *to) -/*[clinic end generated code: output=a36f6399d4b77f6f input=de7a8fc5632bb8f1]*/ -{ - return _Py_bytes_maketrans(frm, to); -} - - -/*[clinic input] -bytes.replace - - old: Py_buffer - new: Py_buffer - count: Py_ssize_t = -1 - Maximum number of occurrences to replace. - -1 (the default value) means replace all occurrences. - / - -Return a copy with all occurrences of substring old replaced by new. - -If the optional argument count is given, only the first count occurrences are -replaced. -[clinic start generated code]*/ - -static PyObject * -bytes_replace_impl(PyBytesObject *self, Py_buffer *old, Py_buffer *new, - Py_ssize_t count) -/*[clinic end generated code: output=994fa588b6b9c104 input=b2fbbf0bf04de8e5]*/ -{ - return stringlib_replace((PyObject *)self, - (const char *)old->buf, old->len, - (const char *)new->buf, new->len, count); -} - -/** End DALKE **/ - - -static PyObject * -bytes_startswith(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_startswith(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - -static PyObject * -bytes_endswith(PyBytesObject *self, PyObject *args) -{ - return _Py_bytes_endswith(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), args); -} - - -/*[clinic input] -bytes.decode - - encoding: str(c_default="NULL") = 'utf-8' - The encoding with which to decode the bytes. - errors: str(c_default="NULL") = 'strict' - The error handling scheme to use for the handling of decoding errors. - The default is 'strict' meaning that decoding errors raise a - UnicodeDecodeError. Other possible values are 'ignore' and 'replace' - as well as any other name registered with codecs.register_error that - can handle UnicodeDecodeErrors. - -Decode the bytes using the codec registered for encoding. -[clinic start generated code]*/ - -static PyObject * -bytes_decode_impl(PyBytesObject *self, const char *encoding, - const char *errors) -/*[clinic end generated code: output=5649a53dde27b314 input=958174769d2a40ca]*/ -{ - return PyUnicode_FromEncodedObject((PyObject*)self, encoding, errors); -} - - -/*[clinic input] -bytes.splitlines - - keepends: bool(accept={int}) = False - -Return a list of the lines in the bytes, breaking at line boundaries. - -Line breaks are not included in the resulting list unless keepends is given and -true. -[clinic start generated code]*/ - -static PyObject * -bytes_splitlines_impl(PyBytesObject *self, int keepends) -/*[clinic end generated code: output=3484149a5d880ffb input=a8b32eb01ff5a5ed]*/ -{ - return stringlib_splitlines( - (PyObject*) self, PyBytes_AS_STRING(self), - PyBytes_GET_SIZE(self), keepends - ); -} - -/*[clinic input] -@classmethod -bytes.fromhex - - string: unicode - / - -Create a bytes object from a string of hexadecimal numbers. - -Spaces between two numbers are accepted. -Example: bytes.fromhex('B9 01EF') -> b'\\xb9\\x01\\xef'. -[clinic start generated code]*/ - -static PyObject * -bytes_fromhex_impl(PyTypeObject *type, PyObject *string) -/*[clinic end generated code: output=0973acc63661bb2e input=bf4d1c361670acd3]*/ -{ - PyObject *result = _PyBytes_FromHex(string, 0); - if (type != &PyBytes_Type && result != NULL) { - Py_SETREF(result, PyObject_CallFunctionObjArgs((PyObject *)type, - result, NULL)); - } - return result; -} - -PyObject* -_PyBytes_FromHex(PyObject *string, int use_bytearray) -{ - char *buf; - Py_ssize_t hexlen, invalid_char; - unsigned int top, bot; - Py_UCS1 *str, *end; - _PyBytesWriter writer; - - _PyBytesWriter_Init(&writer); - writer.use_bytearray = use_bytearray; - - assert(PyUnicode_Check(string)); - if (PyUnicode_READY(string)) - return NULL; - hexlen = PyUnicode_GET_LENGTH(string); - - if (!PyUnicode_IS_ASCII(string)) { - void *data = PyUnicode_DATA(string); - unsigned int kind = PyUnicode_KIND(string); - Py_ssize_t i; - - /* search for the first non-ASCII character */ - for (i = 0; i < hexlen; i++) { - if (PyUnicode_READ(kind, data, i) >= 128) - break; - } - invalid_char = i; - goto error; - } - - assert(PyUnicode_KIND(string) == PyUnicode_1BYTE_KIND); - str = PyUnicode_1BYTE_DATA(string); - - /* This overestimates if there are spaces */ - buf = _PyBytesWriter_Alloc(&writer, hexlen / 2); - if (buf == NULL) - return NULL; - - end = str + hexlen; - while (str < end) { - /* skip over spaces in the input */ - if (Py_ISSPACE(*str)) { - do { - str++; - } while (Py_ISSPACE(*str)); - if (str >= end) - break; - } - - top = _PyLong_DigitValue[*str]; - if (top >= 16) { - invalid_char = str - PyUnicode_1BYTE_DATA(string); - goto error; - } - str++; - - bot = _PyLong_DigitValue[*str]; - if (bot >= 16) { - invalid_char = str - PyUnicode_1BYTE_DATA(string); - goto error; - } - str++; - - *buf++ = (unsigned char)((top << 4) + bot); - } - - return _PyBytesWriter_Finish(&writer, buf); - - error: - PyErr_Format(PyExc_ValueError, - "non-hexadecimal number found in " - "fromhex() arg at position %zd", invalid_char); - _PyBytesWriter_Dealloc(&writer); - return NULL; -} - -PyDoc_STRVAR(hex__doc__, -"B.hex() -> string\n\ -\n\ -Create a string of hexadecimal numbers from a bytes object.\n\ -Example: b'\\xb9\\x01\\xef'.hex() -> 'b901ef'."); - -static PyObject * -bytes_hex(PyBytesObject *self) -{ - char* argbuf = PyBytes_AS_STRING(self); - Py_ssize_t arglen = PyBytes_GET_SIZE(self); - return _Py_strhex(argbuf, arglen); -} - -static PyObject * -bytes_getnewargs(PyBytesObject *v) -{ - return Py_BuildValue("(y#)", v->ob_sval, Py_SIZE(v)); -} - - -static PyMethodDef -bytes_methods[] = { - {"__getnewargs__", (PyCFunction)bytes_getnewargs, METH_NOARGS}, - {"capitalize", (PyCFunction)stringlib_capitalize, METH_NOARGS, - _Py_capitalize__doc__}, - {"center", (PyCFunction)stringlib_center, METH_VARARGS, - _Py_center__doc__}, - {"count", (PyCFunction)bytes_count, METH_VARARGS, - _Py_count__doc__}, - BYTES_DECODE_METHODDEF - {"endswith", (PyCFunction)bytes_endswith, METH_VARARGS, - _Py_endswith__doc__}, - {"expandtabs", (PyCFunction)stringlib_expandtabs, METH_VARARGS | METH_KEYWORDS, - _Py_expandtabs__doc__}, - {"find", (PyCFunction)bytes_find, METH_VARARGS, - _Py_find__doc__}, - BYTES_FROMHEX_METHODDEF - {"hex", (PyCFunction)bytes_hex, METH_NOARGS, hex__doc__}, - {"index", (PyCFunction)bytes_index, METH_VARARGS, _Py_index__doc__}, - {"isalnum", (PyCFunction)stringlib_isalnum, METH_NOARGS, - _Py_isalnum__doc__}, - {"isalpha", (PyCFunction)stringlib_isalpha, METH_NOARGS, - _Py_isalpha__doc__}, - {"isdigit", (PyCFunction)stringlib_isdigit, METH_NOARGS, - _Py_isdigit__doc__}, - {"islower", (PyCFunction)stringlib_islower, METH_NOARGS, - _Py_islower__doc__}, - {"isspace", (PyCFunction)stringlib_isspace, METH_NOARGS, - _Py_isspace__doc__}, - {"istitle", (PyCFunction)stringlib_istitle, METH_NOARGS, - _Py_istitle__doc__}, - {"isupper", (PyCFunction)stringlib_isupper, METH_NOARGS, - _Py_isupper__doc__}, - BYTES_JOIN_METHODDEF - {"ljust", (PyCFunction)stringlib_ljust, METH_VARARGS, _Py_ljust__doc__}, - {"lower", (PyCFunction)stringlib_lower, METH_NOARGS, _Py_lower__doc__}, - BYTES_LSTRIP_METHODDEF - BYTES_MAKETRANS_METHODDEF - BYTES_PARTITION_METHODDEF - BYTES_REPLACE_METHODDEF - {"rfind", (PyCFunction)bytes_rfind, METH_VARARGS, _Py_rfind__doc__}, - {"rindex", (PyCFunction)bytes_rindex, METH_VARARGS, _Py_rindex__doc__}, - {"rjust", (PyCFunction)stringlib_rjust, METH_VARARGS, _Py_rjust__doc__}, - BYTES_RPARTITION_METHODDEF - BYTES_RSPLIT_METHODDEF - BYTES_RSTRIP_METHODDEF - BYTES_SPLIT_METHODDEF - BYTES_SPLITLINES_METHODDEF - {"startswith", (PyCFunction)bytes_startswith, METH_VARARGS, - _Py_startswith__doc__}, - BYTES_STRIP_METHODDEF - {"swapcase", (PyCFunction)stringlib_swapcase, METH_NOARGS, - _Py_swapcase__doc__}, - {"title", (PyCFunction)stringlib_title, METH_NOARGS, _Py_title__doc__}, - BYTES_TRANSLATE_METHODDEF - {"upper", (PyCFunction)stringlib_upper, METH_NOARGS, _Py_upper__doc__}, - {"zfill", (PyCFunction)stringlib_zfill, METH_VARARGS, _Py_zfill__doc__}, - {NULL, NULL} /* sentinel */ -}; - -static PyObject * -bytes_mod(PyObject *self, PyObject *arg) -{ - if (!PyBytes_Check(self)) { - Py_RETURN_NOTIMPLEMENTED; - } - return _PyBytes_FormatEx(PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self), - arg, 0); -} - -static PyNumberMethods bytes_as_number = { - 0, /*nb_add*/ - 0, /*nb_subtract*/ - 0, /*nb_multiply*/ - bytes_mod, /*nb_remainder*/ -}; - -static PyObject * -bytes_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds); - -static PyObject * -bytes_new(PyTypeObject *type, PyObject *args, PyObject *kwds) -{ - PyObject *x = NULL; - const char *encoding = NULL; - const char *errors = NULL; - PyObject *new = NULL; - PyObject *func; - Py_ssize_t size; - static char *kwlist[] = {"source", "encoding", "errors", 0}; - _Py_IDENTIFIER(__bytes__); - - if (type != &PyBytes_Type) - return bytes_subtype_new(type, args, kwds); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|Oss:bytes", kwlist, &x, - &encoding, &errors)) - return NULL; - if (x == NULL) { - if (encoding != NULL || errors != NULL) { - PyErr_SetString(PyExc_TypeError, - "encoding or errors without sequence " - "argument"); - return NULL; - } - return PyBytes_FromStringAndSize(NULL, 0); - } - - if (encoding != NULL) { - /* Encode via the codec registry */ - if (!PyUnicode_Check(x)) { - PyErr_SetString(PyExc_TypeError, - "encoding without a string argument"); - return NULL; - } - new = PyUnicode_AsEncodedString(x, encoding, errors); - if (new == NULL) - return NULL; - assert(PyBytes_Check(new)); - return new; - } - - if (errors != NULL) { - PyErr_SetString(PyExc_TypeError, - PyUnicode_Check(x) ? - "string argument without an encoding" : - "errors without a string argument"); - return NULL; - } - - /* We'd like to call PyObject_Bytes here, but we need to check for an - integer argument before deferring to PyBytes_FromObject, something - PyObject_Bytes doesn't do. */ - func = _PyObject_LookupSpecial(x, &PyId___bytes__); - if (func != NULL) { - new = _PyObject_CallNoArg(func); - Py_DECREF(func); - if (new == NULL) - return NULL; - if (!PyBytes_Check(new)) { - PyErr_Format(PyExc_TypeError, - "__bytes__ returned non-bytes (type %.200s)", - Py_TYPE(new)->tp_name); - Py_DECREF(new); - return NULL; - } - return new; - } - else if (PyErr_Occurred()) - return NULL; - - if (PyUnicode_Check(x)) { - PyErr_SetString(PyExc_TypeError, - "string argument without an encoding"); - return NULL; - } - /* Is it an integer? */ - if (PyIndex_Check(x)) { - size = PyNumber_AsSsize_t(x, PyExc_OverflowError); - if (size == -1 && PyErr_Occurred()) { - if (PyErr_ExceptionMatches(PyExc_OverflowError)) - return NULL; - PyErr_Clear(); /* fall through */ - } - else { - if (size < 0) { - PyErr_SetString(PyExc_ValueError, "negative count"); - return NULL; - } - new = _PyBytes_FromSize(size, 1); - if (new == NULL) - return NULL; - return new; - } - } - - return PyBytes_FromObject(x); -} - -static PyObject* -_PyBytes_FromBuffer(PyObject *x) -{ - PyObject *new; - Py_buffer view; - - if (PyObject_GetBuffer(x, &view, PyBUF_FULL_RO) < 0) - return NULL; - - new = PyBytes_FromStringAndSize(NULL, view.len); - if (!new) - goto fail; - if (PyBuffer_ToContiguous(((PyBytesObject *)new)->ob_sval, - &view, view.len, 'C') < 0) - goto fail; - PyBuffer_Release(&view); - return new; - -fail: - Py_XDECREF(new); - PyBuffer_Release(&view); - return NULL; -} - -#define _PyBytes_FROM_LIST_BODY(x, GET_ITEM) \ - do { \ - PyObject *bytes; \ - Py_ssize_t i; \ - Py_ssize_t value; \ - char *str; \ - PyObject *item; \ - \ - bytes = PyBytes_FromStringAndSize(NULL, Py_SIZE(x)); \ - if (bytes == NULL) \ - return NULL; \ - str = ((PyBytesObject *)bytes)->ob_sval; \ - \ - for (i = 0; i < Py_SIZE(x); i++) { \ - item = GET_ITEM((x), i); \ - value = PyNumber_AsSsize_t(item, NULL); \ - if (value == -1 && PyErr_Occurred()) \ - goto error; \ - \ - if (value < 0 || value >= 256) { \ - PyErr_SetString(PyExc_ValueError, \ - "bytes must be in range(0, 256)"); \ - goto error; \ - } \ - *str++ = (char) value; \ - } \ - return bytes; \ - \ - error: \ - Py_DECREF(bytes); \ - return NULL; \ - } while (0) - -static PyObject* -_PyBytes_FromList(PyObject *x) -{ - _PyBytes_FROM_LIST_BODY(x, PyList_GET_ITEM); -} - -static PyObject* -_PyBytes_FromTuple(PyObject *x) -{ - _PyBytes_FROM_LIST_BODY(x, PyTuple_GET_ITEM); -} - -static PyObject * -_PyBytes_FromIterator(PyObject *it, PyObject *x) -{ - char *str; - Py_ssize_t i, size; - _PyBytesWriter writer; - - /* For iterator version, create a string object and resize as needed */ - size = PyObject_LengthHint(x, 64); - if (size == -1 && PyErr_Occurred()) - return NULL; - - _PyBytesWriter_Init(&writer); - str = _PyBytesWriter_Alloc(&writer, size); - if (str == NULL) - return NULL; - writer.overallocate = 1; - size = writer.allocated; - - /* Run the iterator to exhaustion */ - for (i = 0; ; i++) { - PyObject *item; - Py_ssize_t value; - - /* Get the next item */ - item = PyIter_Next(it); - if (item == NULL) { - if (PyErr_Occurred()) - goto error; - break; - } - - /* Interpret it as an int (__index__) */ - value = PyNumber_AsSsize_t(item, NULL); - Py_DECREF(item); - if (value == -1 && PyErr_Occurred()) - goto error; - - /* Range check */ - if (value < 0 || value >= 256) { - PyErr_SetString(PyExc_ValueError, - "bytes must be in range(0, 256)"); - goto error; - } - - /* Append the byte */ - if (i >= size) { - str = _PyBytesWriter_Resize(&writer, str, size+1); - if (str == NULL) - return NULL; - size = writer.allocated; - } - *str++ = (char) value; - } - - return _PyBytesWriter_Finish(&writer, str); - - error: - _PyBytesWriter_Dealloc(&writer); - return NULL; -} - -PyObject * -PyBytes_FromObject(PyObject *x) -{ - PyObject *it, *result; - - if (x == NULL) { - PyErr_BadInternalCall(); - return NULL; - } - - if (PyBytes_CheckExact(x)) { - Py_INCREF(x); - return x; - } - - /* Use the modern buffer interface */ - if (PyObject_CheckBuffer(x)) - return _PyBytes_FromBuffer(x); - - if (PyList_CheckExact(x)) - return _PyBytes_FromList(x); - - if (PyTuple_CheckExact(x)) - return _PyBytes_FromTuple(x); - - if (!PyUnicode_Check(x)) { - it = PyObject_GetIter(x); - if (it != NULL) { - result = _PyBytes_FromIterator(it, x); - Py_DECREF(it); - return result; - } - } - - PyErr_Format(PyExc_TypeError, - "cannot convert '%.200s' object to bytes", - x->ob_type->tp_name); - return NULL; -} - -static PyObject * -bytes_subtype_new(PyTypeObject *type, PyObject *args, PyObject *kwds) -{ - PyObject *tmp, *pnew; - Py_ssize_t n; - - assert(PyType_IsSubtype(type, &PyBytes_Type)); - tmp = bytes_new(&PyBytes_Type, args, kwds); - if (tmp == NULL) - return NULL; - assert(PyBytes_Check(tmp)); - n = PyBytes_GET_SIZE(tmp); - pnew = type->tp_alloc(type, n); - if (pnew != NULL) { - memcpy(PyBytes_AS_STRING(pnew), - PyBytes_AS_STRING(tmp), n+1); - ((PyBytesObject *)pnew)->ob_shash = - ((PyBytesObject *)tmp)->ob_shash; - } - Py_DECREF(tmp); - return pnew; -} - -PyDoc_STRVAR(bytes_doc, -"bytes(iterable_of_ints) -> bytes\n\ -bytes(string, encoding[, errors]) -> bytes\n\ -bytes(bytes_or_buffer) -> immutable copy of bytes_or_buffer\n\ -bytes(int) -> bytes object of size given by the parameter initialized with null bytes\n\ -bytes() -> empty bytes object\n\ -\n\ -Construct an immutable array of bytes from:\n\ - - an iterable yielding integers in range(256)\n\ - - a text string encoded using the specified encoding\n\ - - any object implementing the buffer API.\n\ - - an integer"); - -static PyObject *bytes_iter(PyObject *seq); - -PyTypeObject PyBytes_Type = { - PyVarObject_HEAD_INIT(&PyType_Type, 0) - "bytes", - PyBytesObject_SIZE, - sizeof(char), - bytes_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_reserved */ - (reprfunc)bytes_repr, /* tp_repr */ - &bytes_as_number, /* tp_as_number */ - &bytes_as_sequence, /* tp_as_sequence */ - &bytes_as_mapping, /* tp_as_mapping */ - (hashfunc)bytes_hash, /* tp_hash */ - 0, /* tp_call */ - bytes_str, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - &bytes_as_buffer, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | - Py_TPFLAGS_BYTES_SUBCLASS, /* tp_flags */ - bytes_doc, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - (richcmpfunc)bytes_richcompare, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - bytes_iter, /* tp_iter */ - 0, /* tp_iternext */ - bytes_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - &PyBaseObject_Type, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - bytes_new, /* tp_new */ - PyObject_Del, /* tp_free */ -}; - -void -PyBytes_Concat(PyObject **pv, PyObject *w) -{ - assert(pv != NULL); - if (*pv == NULL) - return; - if (w == NULL) { - Py_CLEAR(*pv); - return; - } - - if (Py_REFCNT(*pv) == 1 && PyBytes_CheckExact(*pv)) { - /* Only one reference, so we can resize in place */ - Py_ssize_t oldsize; - Py_buffer wb; - - wb.len = -1; - if (PyObject_GetBuffer(w, &wb, PyBUF_SIMPLE) != 0) { - PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s", - Py_TYPE(w)->tp_name, Py_TYPE(*pv)->tp_name); - Py_CLEAR(*pv); - return; - } - - oldsize = PyBytes_GET_SIZE(*pv); - if (oldsize > PY_SSIZE_T_MAX - wb.len) { - PyErr_NoMemory(); - goto error; - } - if (_PyBytes_Resize(pv, oldsize + wb.len) < 0) - goto error; - - memcpy(PyBytes_AS_STRING(*pv) + oldsize, wb.buf, wb.len); - PyBuffer_Release(&wb); - return; - - error: - PyBuffer_Release(&wb); - Py_CLEAR(*pv); - return; - } - - else { - /* Multiple references, need to create new object */ - PyObject *v; - v = bytes_concat(*pv, w); - Py_SETREF(*pv, v); - } -} - -void -PyBytes_ConcatAndDel(PyObject **pv, PyObject *w) -{ - PyBytes_Concat(pv, w); - Py_XDECREF(w); -} - - -/* The following function breaks the notion that bytes are immutable: - it changes the size of a bytes object. We get away with this only if there - is only one module referencing the object. You can also think of it - as creating a new bytes object and destroying the old one, only - more efficiently. In any case, don't use this if the bytes object may - already be known to some other part of the code... - Note that if there's not enough memory to resize the bytes object, the - original bytes object at *pv is deallocated, *pv is set to NULL, an "out of - memory" exception is set, and -1 is returned. Else (on success) 0 is - returned, and the value in *pv may or may not be the same as on input. - As always, an extra byte is allocated for a trailing \0 byte (newsize - does *not* include that), and a trailing \0 byte is stored. -*/ - -int -_PyBytes_Resize(PyObject **pv, Py_ssize_t newsize) -{ - PyObject *v; - PyBytesObject *sv; - v = *pv; - if (!PyBytes_Check(v) || newsize < 0) { - goto error; - } - if (Py_SIZE(v) == newsize) { - /* return early if newsize equals to v->ob_size */ - return 0; - } - if (Py_REFCNT(v) != 1) { - goto error; - } - /* XXX UNREF/NEWREF interface should be more symmetrical */ - _Py_DEC_REFTOTAL; - _Py_ForgetReference(v); - *pv = (PyObject *) - PyObject_REALLOC(v, PyBytesObject_SIZE + newsize); - if (*pv == NULL) { - PyObject_Del(v); - PyErr_NoMemory(); - return -1; - } - _Py_NewReference(*pv); - sv = (PyBytesObject *) *pv; - Py_SIZE(sv) = newsize; - sv->ob_sval[newsize] = '\0'; - sv->ob_shash = -1; /* invalidate cached hash value */ - return 0; -error: - *pv = 0; - Py_DECREF(v); - PyErr_BadInternalCall(); - return -1; -} - -void -PyBytes_Fini(void) -{ - int i; - for (i = 0; i < UCHAR_MAX + 1; i++) - Py_CLEAR(characters[i]); - Py_CLEAR(nullstring); -} - -/*********************** Bytes Iterator ****************************/ - -typedef struct { - PyObject_HEAD - Py_ssize_t it_index; - PyBytesObject *it_seq; /* Set to NULL when iterator is exhausted */ -} striterobject; - -static void -striter_dealloc(striterobject *it) -{ - _PyObject_GC_UNTRACK(it); - Py_XDECREF(it->it_seq); - PyObject_GC_Del(it); -} - -static int -striter_traverse(striterobject *it, visitproc visit, void *arg) -{ - Py_VISIT(it->it_seq); - return 0; -} - -static PyObject * -striter_next(striterobject *it) -{ - PyBytesObject *seq; - PyObject *item; - - assert(it != NULL); - seq = it->it_seq; - if (seq == NULL) - return NULL; - assert(PyBytes_Check(seq)); - - if (it->it_index < PyBytes_GET_SIZE(seq)) { - item = PyLong_FromLong( - (unsigned char)seq->ob_sval[it->it_index]); - if (item != NULL) - ++it->it_index; - return item; - } - - it->it_seq = NULL; - Py_DECREF(seq); - return NULL; -} - -static PyObject * -striter_len(striterobject *it) -{ - Py_ssize_t len = 0; - if (it->it_seq) - len = PyBytes_GET_SIZE(it->it_seq) - it->it_index; - return PyLong_FromSsize_t(len); -} - -PyDoc_STRVAR(length_hint_doc, - "Private method returning an estimate of len(list(it))."); - -static PyObject * -striter_reduce(striterobject *it) -{ - if (it->it_seq != NULL) { - return Py_BuildValue("N(O)n", _PyObject_GetBuiltin("iter"), - it->it_seq, it->it_index); - } else { - return Py_BuildValue("N(())", _PyObject_GetBuiltin("iter")); - } -} - -PyDoc_STRVAR(reduce_doc, "Return state information for pickling."); - -static PyObject * -striter_setstate(striterobject *it, PyObject *state) -{ - Py_ssize_t index = PyLong_AsSsize_t(state); - if (index == -1 && PyErr_Occurred()) - return NULL; - if (it->it_seq != NULL) { - if (index < 0) - index = 0; - else if (index > PyBytes_GET_SIZE(it->it_seq)) - index = PyBytes_GET_SIZE(it->it_seq); /* iterator exhausted */ - it->it_index = index; - } - Py_RETURN_NONE; -} - -PyDoc_STRVAR(setstate_doc, "Set state information for unpickling."); - -static PyMethodDef striter_methods[] = { - {"__length_hint__", (PyCFunction)striter_len, METH_NOARGS, - length_hint_doc}, - {"__reduce__", (PyCFunction)striter_reduce, METH_NOARGS, - reduce_doc}, - {"__setstate__", (PyCFunction)striter_setstate, METH_O, - setstate_doc}, - {NULL, NULL} /* sentinel */ -}; - -PyTypeObject PyBytesIter_Type = { - PyVarObject_HEAD_INIT(&PyType_Type, 0) - "bytes_iterator", /* tp_name */ - sizeof(striterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)striter_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_reserved */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ - 0, /* tp_doc */ - (traverseproc)striter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - PyObject_SelfIter, /* tp_iter */ - (iternextfunc)striter_next, /* tp_iternext */ - striter_methods, /* tp_methods */ - 0, -}; - -static PyObject * -bytes_iter(PyObject *seq) -{ - striterobject *it; - - if (!PyBytes_Check(seq)) { - PyErr_BadInternalCall(); - return NULL; - } - it = PyObject_GC_New(striterobject, &PyBytesIter_Type); - if (it == NULL) - return NULL; - it->it_index = 0; - Py_INCREF(seq); - it->it_seq = (PyBytesObject *)seq; - _PyObject_GC_TRACK(it); - return (PyObject *)it; -} - - -/* _PyBytesWriter API */ - -#ifdef MS_WINDOWS - /* On Windows, overallocate by 50% is the best factor */ -# define OVERALLOCATE_FACTOR 2 -#else - /* On Linux, overallocate by 25% is the best factor */ -# define OVERALLOCATE_FACTOR 4 -#endif - -void -_PyBytesWriter_Init(_PyBytesWriter *writer) -{ - /* Set all attributes before small_buffer to 0 */ - memset(writer, 0, offsetof(_PyBytesWriter, small_buffer)); -#ifdef Py_DEBUG - memset(writer->small_buffer, 0xCB, sizeof(writer->small_buffer)); -#endif -} - -void -_PyBytesWriter_Dealloc(_PyBytesWriter *writer) -{ - Py_CLEAR(writer->buffer); -} - -Py_LOCAL_INLINE(char*) -_PyBytesWriter_AsString(_PyBytesWriter *writer) -{ - if (writer->use_small_buffer) { - assert(writer->buffer == NULL); - return writer->small_buffer; - } - else if (writer->use_bytearray) { - assert(writer->buffer != NULL); - return PyByteArray_AS_STRING(writer->buffer); - } - else { - assert(writer->buffer != NULL); - return PyBytes_AS_STRING(writer->buffer); - } -} - -Py_LOCAL_INLINE(Py_ssize_t) -_PyBytesWriter_GetSize(_PyBytesWriter *writer, char *str) -{ - char *start = _PyBytesWriter_AsString(writer); - assert(str != NULL); - assert(str >= start); - assert(str - start <= writer->allocated); - return str - start; -} - -Py_LOCAL_INLINE(void) -_PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str) -{ -#ifdef Py_DEBUG - char *start, *end; - - if (writer->use_small_buffer) { - assert(writer->buffer == NULL); - } - else { - assert(writer->buffer != NULL); - if (writer->use_bytearray) - assert(PyByteArray_CheckExact(writer->buffer)); - else - assert(PyBytes_CheckExact(writer->buffer)); - assert(Py_REFCNT(writer->buffer) == 1); - } - - if (writer->use_bytearray) { - /* bytearray has its own overallocation algorithm, - writer overallocation must be disabled */ - assert(!writer->overallocate); - } - - assert(0 <= writer->allocated); - assert(0 <= writer->min_size && writer->min_size <= writer->allocated); - /* the last byte must always be null */ - start = _PyBytesWriter_AsString(writer); - assert(start[writer->allocated] == 0); - - end = start + writer->allocated; - assert(str != NULL); - assert(start <= str && str <= end); -#endif -} - -void* -_PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size) -{ - Py_ssize_t allocated, pos; - - _PyBytesWriter_CheckConsistency(writer, str); - assert(writer->allocated < size); - - allocated = size; - if (writer->overallocate - && allocated <= (PY_SSIZE_T_MAX - allocated / OVERALLOCATE_FACTOR)) { - /* overallocate to limit the number of realloc() */ - allocated += allocated / OVERALLOCATE_FACTOR; - } - - pos = _PyBytesWriter_GetSize(writer, str); - if (!writer->use_small_buffer) { - if (writer->use_bytearray) { - if (PyByteArray_Resize(writer->buffer, allocated)) - goto error; - /* writer->allocated can be smaller than writer->buffer->ob_alloc, - but we cannot use ob_alloc because bytes may need to be moved - to use the whole buffer. bytearray uses an internal optimization - to avoid moving or copying bytes when bytes are removed at the - beginning (ex: del bytearray[:1]). */ - } - else { - if (_PyBytes_Resize(&writer->buffer, allocated)) - goto error; - } - } - else { - /* convert from stack buffer to bytes object buffer */ - assert(writer->buffer == NULL); - - if (writer->use_bytearray) - writer->buffer = PyByteArray_FromStringAndSize(NULL, allocated); - else - writer->buffer = PyBytes_FromStringAndSize(NULL, allocated); - if (writer->buffer == NULL) - goto error; - - if (pos != 0) { - char *dest; - if (writer->use_bytearray) - dest = PyByteArray_AS_STRING(writer->buffer); - else - dest = PyBytes_AS_STRING(writer->buffer); - memcpy(dest, - writer->small_buffer, - pos); - } - - writer->use_small_buffer = 0; -#ifdef Py_DEBUG - memset(writer->small_buffer, 0xDB, sizeof(writer->small_buffer)); -#endif - } - writer->allocated = allocated; - - str = _PyBytesWriter_AsString(writer) + pos; - _PyBytesWriter_CheckConsistency(writer, str); - return str; - -error: - _PyBytesWriter_Dealloc(writer); - return NULL; -} - -void* -_PyBytesWriter_Prepare(_PyBytesWriter *writer, void *str, Py_ssize_t size) -{ - Py_ssize_t new_min_size; - - _PyBytesWriter_CheckConsistency(writer, str); - assert(size >= 0); - - if (size == 0) { - /* nothing to do */ - return str; - } - - if (writer->min_size > PY_SSIZE_T_MAX - size) { - PyErr_NoMemory(); - _PyBytesWriter_Dealloc(writer); - return NULL; - } - new_min_size = writer->min_size + size; - - if (new_min_size > writer->allocated) - str = _PyBytesWriter_Resize(writer, str, new_min_size); - - writer->min_size = new_min_size; - return str; -} - -/* Allocate the buffer to write size bytes. - Return the pointer to the beginning of buffer data. - Raise an exception and return NULL on error. */ -void* -_PyBytesWriter_Alloc(_PyBytesWriter *writer, Py_ssize_t size) -{ - /* ensure that _PyBytesWriter_Alloc() is only called once */ - assert(writer->min_size == 0 && writer->buffer == NULL); - assert(size >= 0); - - writer->use_small_buffer = 1; -#ifdef Py_DEBUG - writer->allocated = sizeof(writer->small_buffer) - 1; - /* In debug mode, don't use the full small buffer because it is less - efficient than bytes and bytearray objects to detect buffer underflow - and buffer overflow. Use 10 bytes of the small buffer to test also - code using the smaller buffer in debug mode. - - Don't modify the _PyBytesWriter structure (use a shorter small buffer) - in debug mode to also be able to detect stack overflow when running - tests in debug mode. The _PyBytesWriter is large (more than 512 bytes), - if Py_EnterRecursiveCall() is not used in deep C callback, we may hit a - stack overflow. */ - writer->allocated = Py_MIN(writer->allocated, 10); - /* _PyBytesWriter_CheckConsistency() requires the last byte to be 0, - to detect buffer overflow */ - writer->small_buffer[writer->allocated] = 0; -#else - writer->allocated = sizeof(writer->small_buffer); -#endif - return _PyBytesWriter_Prepare(writer, writer->small_buffer, size); -} - -PyObject * -_PyBytesWriter_Finish(_PyBytesWriter *writer, void *str) -{ - Py_ssize_t size; - PyObject *result; - - _PyBytesWriter_CheckConsistency(writer, str); - - size = _PyBytesWriter_GetSize(writer, str); - if (size == 0 && !writer->use_bytearray) { - Py_CLEAR(writer->buffer); - /* Get the empty byte string singleton */ - result = PyBytes_FromStringAndSize(NULL, 0); - } - else if (writer->use_small_buffer) { - if (writer->use_bytearray) { - result = PyByteArray_FromStringAndSize(writer->small_buffer, size); - } - else { - result = PyBytes_FromStringAndSize(writer->small_buffer, size); - } - } - else { - result = writer->buffer; - writer->buffer = NULL; - - if (size != writer->allocated) { - if (writer->use_bytearray) { - if (PyByteArray_Resize(result, size)) { - Py_DECREF(result); - return NULL; - } - } - else { - if (_PyBytes_Resize(&result, size)) { - assert(result == NULL); - return NULL; - } - } - } - } - return result; -} - -void* -_PyBytesWriter_WriteBytes(_PyBytesWriter *writer, void *ptr, - const void *bytes, Py_ssize_t size) -{ - char *str = (char *)ptr; - - str = _PyBytesWriter_Prepare(writer, str, size); - if (str == NULL) - return NULL; - - memcpy(str, bytes, size); - str += size; - - return str; -} diff --git a/Objects/classobject.c b/Objects/classobject.c index 5b64578..02d7cfd 100644 --- a/Objects/classobject.c +++ b/Objects/classobject.c @@ -1009,7 +1009,7 @@ instance_hash(PyInstanceObject *inst) Py_DECREF(func); if (res == NULL) return -1; - if (PyInt_Check(res) || PyLong_Check(res)) + if (_PyAnyInt_Check(res)) /* This already converts a -1 result to -2. */ outcome = Py_TYPE(res)->tp_hash(res); else { diff --git a/Objects/codeobject.c b/Objects/codeobject.c index a66aa69..d50e4c6 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -423,10 +423,9 @@ _PyCode_ConstantKey(PyObject *op) /* Py_None is a singleton */ if (op == Py_None - || PyInt_CheckExact(op) - || PyLong_CheckExact(op) - || PyBool_Check(op) - || PyBytes_CheckExact(op) + || _PyAnyInt_CheckExact(op) + || PyBool_Check(op) + || PyBytes_CheckExact(op) #ifdef Py_USING_UNICODE || PyUnicode_CheckExact(op) #endif diff --git a/Objects/complexobject.c b/Objects/complexobject.c index aaefa2d..871eea3 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -796,7 +796,7 @@ complex_richcompare(PyObject *v, PyObject *w, int op) * NotImplemented. Only comparisons with core numeric types raise * TypeError. */ - if (PyInt_Check(w) || PyLong_Check(w) || + if (_PyAnyInt_Check(w) || PyFloat_Check(w) || PyComplex_Check(w)) { PyErr_SetString(PyExc_TypeError, "no ordering relation is defined " @@ -809,7 +809,7 @@ complex_richcompare(PyObject *v, PyObject *w, int op) assert(PyComplex_Check(v)); TO_COMPLEX(v, i); - if (PyInt_Check(w) || PyLong_Check(w)) { + if (_PyAnyInt_Check(w)) { /* Check for 0.0 imaginary part first to avoid the rich * comparison when possible. */ diff --git a/Objects/enumobject.c b/Objects/enumobject.c index 8f86a5b..73b656b 100644 --- a/Objects/enumobject.c +++ b/Objects/enumobject.c @@ -31,7 +31,7 @@ enum_new(PyTypeObject *type, PyObject *args, PyObject *kwds) Py_DECREF(en); return NULL; } - assert(PyInt_Check(start) || PyLong_Check(start)); + assert(_PyAnyInt_Check(start)); en->en_index = PyInt_AsSsize_t(start); if (en->en_index == -1 && PyErr_Occurred()) { PyErr_Clear(); diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 37dd67e..5954d39 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -428,7 +428,7 @@ float_richcompare(PyObject *v, PyObject *w, int op) j = PyFloat_AS_DOUBLE(w); else if (!Py_IS_FINITE(i)) { - if (PyInt_Check(w) || PyLong_Check(w)) + if (_PyAnyInt_Check(w)) /* If i is an infinity, its magnitude exceeds any * finite integer, so it doesn't matter which int we * compare i with. If i is a NaN, similarly. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 1758745..4c91dd0 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -118,6 +118,10 @@ frame_setlineno(PyFrameObject *f, PyObject* p_new_lineno) int blockstack_top = 0; /* (ditto) */ unsigned char setup_op = 0; /* (ditto) */ + if (p_new_lineno == NULL) { + PyErr_SetString(PyExc_AttributeError, "cannot delete attribute"); + return -1; + } /* f_lineno must be an integer. */ if (!PyInt_Check(p_new_lineno)) { PyErr_SetString(PyExc_ValueError, diff --git a/Objects/intobject.c b/Objects/intobject.c index 3ab00af..9b27c35 100644 --- a/Objects/intobject.c +++ b/Objects/intobject.c @@ -358,7 +358,7 @@ PyInt_FromString(char *s, char **pend, int base) if ((base != 0 && base < 2) || base > 36) { PyErr_SetString(PyExc_ValueError, - "int() base must be >= 2 and <= 36"); + "int() base must be >= 2 and <= 36, or 0"); return NULL; } diff --git a/Objects/longobject.c b/Objects/longobject.c index 5d6ce70..f40ad7a 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -1722,7 +1722,7 @@ PyLong_FromString(char *str, char **pend, int base) if ((base != 0 && base < 2) || base > 36) { PyErr_SetString(PyExc_ValueError, - "long() arg 2 must be >= 2 and <= 36"); + "long() base must be >= 2 and <= 36, or 0"); return NULL; } while (*str != '\0' && isspace(Py_CHARMASK(*str))) diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index f2fed30..fd48ff1 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -221,7 +221,7 @@ The name must be a string; the optional doc argument can have any type."); PyTypeObject PyModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "module", /* tp_name */ - sizeof(PyModuleObject), /* tp_size */ + sizeof(PyModuleObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)module_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 9adcff7..0778c85 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -1413,6 +1413,38 @@ pool_is_in_list(const poolp target, poolp list) #endif /* Py_DEBUG */ +static void * +_PyMem_Malloc(size_t nbytes) +{ + if (nbytes > (size_t)PY_SSIZE_T_MAX) { + return NULL; + } + if (nbytes == 0) { + nbytes = 1; + } + return malloc(nbytes); +} + +static void * +_PyMem_Realloc(void *p, size_t nbytes) +{ + if (nbytes > (size_t)PY_SSIZE_T_MAX) { + return NULL; + } + if (nbytes == 0) { + nbytes = 1; + } + return realloc(p, nbytes); +} + + +static void +_PyMem_Free(void *p) +{ + free(p); +} + + /* Let S = sizeof(size_t). The debug malloc asks for 4*S extra bytes and fills them with useful stuff, here calling the underlying malloc's result p: @@ -1479,7 +1511,7 @@ _PyObject_DebugCheckAddress(const void *p) /* generic debug memory api, with an "id" to identify the API in use */ void * -_PyObject_DebugMallocApi(char id, size_t nbytes) +_PyObject_DebugMallocApi(char api, size_t nbytes) { uchar *p; /* base address of malloc'ed block */ uchar *tail; /* p + 2*SST + nbytes == pointer to tail pad bytes */ @@ -1491,13 +1523,18 @@ _PyObject_DebugMallocApi(char id, size_t nbytes) /* overflow: can't represent total as a size_t */ return NULL; - p = (uchar *)PyObject_Malloc(total); + if (api == _PYMALLOC_OBJ_ID) { + p = (uchar *)PyObject_Malloc(total); + } + else { + p = (uchar *)_PyMem_Malloc(total); + } if (p == NULL) return NULL; - /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */ + /* at p, write size (SST bytes), api (1 byte), pad (SST-1 bytes) */ write_size_t(p, nbytes); - p[SST] = (uchar)id; + p[SST] = (uchar)api; memset(p + SST + 1 , FORBIDDENBYTE, SST-1); if (nbytes > 0) @@ -1529,7 +1566,12 @@ _PyObject_DebugFreeApi(char api, void *p) nbytes += 4*SST; if (nbytes > 0) memset(q, DEADBYTE, nbytes); - PyObject_Free(q); + if (api == _PYMALLOC_OBJ_ID) { + PyObject_Free(q); + } + else { + _PyMem_Free(q); + } } void * @@ -1561,7 +1603,12 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes) * case we didn't get the chance to mark the old memory with DEADBYTE, * but we live with that. */ - q = (uchar *)PyObject_Realloc(q - 2*SST, total); + if (api == _PYMALLOC_OBJ_ID) { + q = (uchar *)PyObject_Realloc(q - 2*SST, total); + } + else { + q = (uchar *)_PyMem_Realloc(q - 2*SST, total); + } if (q == NULL) { if (nbytes <= original_nbytes) { /* bpo-31626: the memset() above expects that realloc never fails diff --git a/Objects/setobject.c b/Objects/setobject.c index 154be43..31da3db 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1710,8 +1710,10 @@ set_symmetric_difference(PySetObject *so, PyObject *other) if (otherset == NULL) return NULL; rv = set_symmetric_difference_update(otherset, (PyObject *)so); - if (rv == NULL) + if (rv == NULL) { + Py_DECREF(otherset); return NULL; + } Py_DECREF(rv); return (PyObject *)otherset; } diff --git a/Objects/sliceobject.c b/Objects/sliceobject.c index c66e057..dc18211 100644 --- a/Objects/sliceobject.c +++ b/Objects/sliceobject.c @@ -107,20 +107,20 @@ PySlice_GetIndices(PySliceObject *r, Py_ssize_t length, if (r->step == Py_None) { *step = 1; } else { - if (!PyInt_Check(r->step) && !PyLong_Check(r->step)) return -1; + if (!_PyAnyInt_Check(r->step)) return -1; *step = PyInt_AsSsize_t(r->step); } if (r->start == Py_None) { *start = *step < 0 ? length-1 : 0; } else { - if (!PyInt_Check(r->start) && !PyLong_Check(r->step)) return -1; + if (!_PyAnyInt_Check(r->start)) return -1; *start = PyInt_AsSsize_t(r->start); if (*start < 0) *start += length; } if (r->stop == Py_None) { *stop = *step < 0 ? -1 : length; } else { - if (!PyInt_Check(r->stop) && !PyLong_Check(r->step)) return -1; + if (!_PyAnyInt_Check(r->stop)) return -1; *stop = PyInt_AsSsize_t(r->stop); if (*stop < 0) *stop += length; } diff --git a/Objects/stringobject.c b/Objects/stringobject.c index 59d22e7..c47d32f 100644 --- a/Objects/stringobject.c +++ b/Objects/stringobject.c @@ -360,9 +360,15 @@ PyString_FromFormatV(const char *format, va_list vargs) break; case 's': p = va_arg(vargs, char*); - i = strlen(p); - if (n > 0 && i > n) - i = n; + if (n <= 0) { + i = strlen(p); + } + else { + i = 0; + while (i < n && p[i]) { + i++; + } + } Py_MEMCPY(s, p, i); s += i; break; @@ -3893,13 +3899,31 @@ _PyString_Resize(PyObject **pv, Py_ssize_t newsize) register PyObject *v; register PyStringObject *sv; v = *pv; - if (!PyString_Check(v) || Py_REFCNT(v) != 1 || newsize < 0 || - PyString_CHECK_INTERNED(v)) { + if (!PyString_Check(v) || newsize < 0) { + *pv = 0; + Py_DECREF(v); + PyErr_BadInternalCall(); + return -1; + } + if (Py_SIZE(v) == 0) { + if (newsize == 0) { + return 0; + } + *pv = PyString_FromStringAndSize(NULL, newsize); + Py_DECREF(v); + return (*pv == NULL) ? -1 : 0; + } + if (Py_REFCNT(v) != 1 || PyString_CHECK_INTERNED(v)) { *pv = 0; Py_DECREF(v); PyErr_BadInternalCall(); return -1; } + if (newsize == 0) { + *pv = PyString_FromStringAndSize(NULL, 0); + Py_DECREF(v); + return (*pv == NULL) ? -1 : 0; + } /* XXX UNREF/NEWREF interface should be more symmetrical */ _Py_DEC_REFTOTAL; _Py_ForgetReference(v); @@ -4504,7 +4528,7 @@ PyString_Format(PyObject *format, PyObject *args) if (PyNumber_Check(v)) { PyObject *iobj=NULL; - if (PyInt_Check(v) || (PyLong_Check(v))) { + if (_PyAnyInt_Check(v)) { iobj = v; Py_INCREF(iobj); } diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 685c545..844fb00 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2082,6 +2082,9 @@ type_init(PyObject *cls, PyObject *args, PyObject *kwds) /* Call object.__init__(self) now. */ /* XXX Could call super(type, cls).__init__() but what's the point? */ args = PyTuple_GetSlice(args, 0, 0); + if (args == NULL) { + return -1; + } res = object_init(cls, args, NULL); Py_DECREF(args); return res; @@ -2608,6 +2611,7 @@ type_getattro(PyTypeObject *type, PyObject *name) PyTypeObject *metatype = Py_TYPE(type); PyObject *meta_attribute, *attribute; descrgetfunc meta_get; + PyObject* res; if (!PyString_Check(name)) { PyErr_Format(PyExc_TypeError, @@ -2629,6 +2633,7 @@ type_getattro(PyTypeObject *type, PyObject *name) meta_attribute = _PyType_Lookup(metatype, name); if (meta_attribute != NULL) { + Py_INCREF(meta_attribute); meta_get = Py_TYPE(meta_attribute)->tp_descr_get; if (meta_get != NULL && PyDescr_IsData(meta_attribute)) { @@ -2636,10 +2641,11 @@ type_getattro(PyTypeObject *type, PyObject *name) * writes. Assume the attribute is not overridden in * type's tp_dict (and bases): call the descriptor now. */ - return meta_get(meta_attribute, (PyObject *)type, - (PyObject *)metatype); + res = meta_get(meta_attribute, (PyObject *)type, + (PyObject *)metatype); + Py_DECREF(meta_attribute); + return res; } - Py_INCREF(meta_attribute); } /* No data descriptor found on metatype. Look in tp_dict of this @@ -2647,18 +2653,21 @@ type_getattro(PyTypeObject *type, PyObject *name) attribute = _PyType_Lookup(type, name); if (attribute != NULL) { /* Implement descriptor functionality, if any */ - descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get; + descrgetfunc local_get; + Py_INCREF(attribute); + local_get = Py_TYPE(attribute)->tp_descr_get; Py_XDECREF(meta_attribute); if (local_get != NULL) { /* NULL 2nd argument indicates the descriptor was * found on the target object itself (or a base) */ - return local_get(attribute, (PyObject *)NULL, - (PyObject *)type); + res = local_get(attribute, (PyObject *)NULL, + (PyObject *)type); + Py_DECREF(attribute); + return res; } - Py_INCREF(attribute); return attribute; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index d011f7d..a859fa0 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -421,10 +421,27 @@ int _PyUnicode_Resize(PyUnicodeObject **unicode, Py_ssize_t length) return -1; } v = *unicode; - if (v == NULL || !PyUnicode_Check(v) || Py_REFCNT(v) != 1 || length < 0) { + if (v == NULL || !PyUnicode_Check(v) || length < 0) { PyErr_BadInternalCall(); return -1; } + if (v->length == 0) { + if (length == 0) { + return 0; + } + *unicode = _PyUnicode_New(length); + Py_DECREF(v); + return (*unicode == NULL) ? -1 : 0; + } + if (Py_REFCNT(v) != 1) { + PyErr_BadInternalCall(); + return -1; + } + if (length == 0) { + *unicode = _PyUnicode_New(0); + Py_DECREF(v); + return (*unicode == NULL) ? -1 : 0; + } /* Resizing unicode_empty and single character objects is not possible since these are being shared. We simply return a fresh @@ -2950,7 +2967,7 @@ PyObject *PyUnicode_DecodeUnicodeEscape(const char *s, if (ucnhash_CAPI == NULL) goto ucnhashError; } - if (*s == '{') { + if (s < end && *s == '{') { const char *start = s+1; /* look for the closing brace */ while (*s != '}' && s < end) @@ -4222,7 +4239,7 @@ PyObject *PyUnicode_DecodeCharmap(const char *s, p = PyUnicode_AS_UNICODE(v) + oldpos; } value -= 0x10000; - *p++ = 0xD800 | (value >> 10); + *p++ = 0xD800 | (Py_UNICODE)(value >> 10); *p++ = 0xDC00 | (value & 0x3FF); extrachars -= 2; } @@ -8628,7 +8645,7 @@ PyObject *PyUnicode_Format(PyObject *format, if (PyNumber_Check(v)) { PyObject *iobj=NULL; - if (PyInt_Check(v) || (PyLong_Check(v))) { + if (_PyAnyInt_Check(v)) { iobj = v; Py_INCREF(iobj); } diff --git a/PC/VS9.0/build.bat b/PC/VS9.0/build.bat index 6225fdf..f489551 100644 --- a/PC/VS9.0/build.bat +++ b/PC/VS9.0/build.bat @@ -76,9 +76,10 @@ if '%build_tkinter%'=='true' ( )
if '%build_tkinter%'=='true' (
if not exist "%tcltkdir%\bin\tcl85%tcl_dbg_ext%.dll" (
- @rem all and install need to be separate invocations, otherwise nmakehlp is not found on install
+ @rem clean, all and install need to be separate invocations, otherwise nmakehlp is not found on install
pushd "%tcldir%\win"
- nmake -f makefile.vc MACHINE=%machine% DEBUG=%debug_flag% INSTALLDIR="%tcltkdir%" clean all
+ nmake -f makefile.vc MACHINE=%machine% DEBUG=%debug_flag% INSTALLDIR="%tcltkdir%" clean
+ nmake -f makefile.vc MACHINE=%machine% DEBUG=%debug_flag% INSTALLDIR="%tcltkdir%" all
nmake -f makefile.vc MACHINE=%machine% DEBUG=%debug_flag% INSTALLDIR="%tcltkdir%" install
popd
)
diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops index 3578ab9..1f5f282 100644 --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -82,7 +82,7 @@ />
<UserMacro
Name="opensslDir"
- Value="$(externalsDir)\openssl-1.0.2k"
+ Value="$(externalsDir)\openssl-1.0.2o"
/>
<UserMacro
Name="tcltkDir"
diff --git a/PC/VS9.0/readme.txt b/PC/VS9.0/readme.txt index 55dda78..a4024d5 100644 --- a/PC/VS9.0/readme.txt +++ b/PC/VS9.0/readme.txt @@ -132,7 +132,7 @@ _ssl Get the source code through
- svn export http://svn.python.org/projects/external/openssl-1.0.2k
+ svn export http://svn.python.org/projects/external/openssl-1.0.2o
** NOTE: if you use the PCbuild\get_externals.bat approach for
obtaining external sources then you don't need to manually get the source
@@ -321,6 +321,10 @@ msierror(int status) code = MsiRecordGetInteger(err, 1); /* XXX code */ if (MsiFormatRecord(0, err, res, &size) == ERROR_MORE_DATA) { res = malloc(size+1); + if (res == NULL) { + MsiCloseHandle(err); + return PyErr_NoMemory(); + } MsiFormatRecord(0, err, res, &size); res[size]='\0'; } @@ -544,6 +548,9 @@ summary_getproperty(msiobj* si, PyObject *args) &fval, sval, &ssize); if (status == ERROR_MORE_DATA) { sval = malloc(ssize); + if (sval == NULL) { + return PyErr_NoMemory(); + } status = MsiSummaryInfoGetProperty(si->h, field, &type, &ival, &fval, sval, &ssize); } diff --git a/PC/_subprocess.c b/PC/_subprocess.c index fc9aaa4..66940ff 100644 --- a/PC/_subprocess.c +++ b/PC/_subprocess.c @@ -381,7 +381,7 @@ getenvironment(PyObject* environment) } totalsize = (p - PyString_AS_STRING(out)) + ksize + 1 + vsize + 1 + 1; - if (totalsize > PyString_GET_SIZE(out)) { + if (totalsize > (size_t)PyString_GET_SIZE(out)) { size_t offset = p - PyString_AS_STRING(out); if (_PyString_Resize(&out, totalsize + 1024)) goto exit; diff --git a/PC/_winreg.c b/PC/_winreg.c index e139cce..f0f8df3 100644 --- a/PC/_winreg.c +++ b/PC/_winreg.c @@ -641,7 +641,7 @@ PyHKEY_AsHKEY(PyObject *ob, HKEY *pHANDLE, BOOL bNoneOK) PyHKEYObject *pH = (PyHKEYObject *)ob; *pHANDLE = pH->hkey; } - else if (PyInt_Check(ob) || PyLong_Check(ob)) { + else if (_PyAnyInt_Check(ob)) { /* We also support integers */ PyErr_Clear(); *pHANDLE = (HKEY)PyLong_AsVoidPtr(ob); @@ -753,8 +753,7 @@ Py2Reg(PyObject *value, DWORD typ, BYTE **retDataBuf, DWORD *retDataSize) Py_ssize_t i,j; switch (typ) { case REG_DWORD: - if (value != Py_None && - !(PyInt_Check(value) || PyLong_Check(value))) + if (value != Py_None && !_PyAnyInt_Check(value)) return FALSE; *retDataBuf = (BYTE *)PyMem_NEW(DWORD, 1); if (*retDataBuf==NULL){ diff --git a/PC/pyconfig.h b/PC/pyconfig.h index 5d1c90a..5fa6398 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -226,11 +226,6 @@ typedef int pid_t; #define Py_IS_FINITE(X) _finite(X) #define copysign _copysign -/* VS 2010 and above already defines hypot as _hypot */ -#if _MSC_VER < 1600 -#define hypot _hypot -#endif - #endif /* _MSC_VER */ /* define some ANSI types that are not defined in earlier Win headers */ @@ -282,7 +277,6 @@ typedef int pid_t; #endif #define COMPILER "[gcc]" -#define hypot _hypot #define PY_LONG_LONG long long #define PY_LLONG_MIN LLONG_MIN #define PY_LLONG_MAX LLONG_MAX diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index b5cf00c..042cff6 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -47,7 +47,7 @@ rem files in both this dir and PC\VS9.0 set libraries=
set libraries=%libraries% bzip2-1.0.6
if NOT "%IncludeBsddb%"=="false" set libraries=%libraries% bsddb-4.7.25.0
-if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2o
+if NOT "%IncludeSSL%"=="false" set libraries=%libraries% openssl-1.0.2q
set libraries=%libraries% sqlite-3.14.2.0
if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tcl-8.5.19.0
if NOT "%IncludeTkinter%"=="false" set libraries=%libraries% tk-8.5.19.0
diff --git a/PCbuild/libeay.vcxproj b/PCbuild/libeay.vcxproj index 1eb2917..4e46ede 100644 --- a/PCbuild/libeay.vcxproj +++ b/PCbuild/libeay.vcxproj @@ -533,6 +533,7 @@ <!--
<ClCompile Include="$(opensslDir)crypto\fips_ers.c" />
-->
+ <ClCompile Include="$(opensslDir)crypto\getenv.c" />
<ClCompile Include="$(opensslDir)crypto\hmac\hm_ameth.c" />
<ClCompile Include="$(opensslDir)crypto\hmac\hm_pmeth.c" />
<ClCompile Include="$(opensslDir)crypto\hmac\hmac.c" />
@@ -725,9 +726,7 @@ <ClCompile Include="$(opensslDir)crypto\ui\ui_lib.c" />
<ClCompile Include="$(opensslDir)crypto\ui\ui_openssl.c" />
<ClCompile Include="$(opensslDir)crypto\ui\ui_util.c" />
- <!--
<ClCompile Include="$(opensslDir)crypto\uid.c" />
- -->
<ClCompile Include="$(opensslDir)crypto\whrlpool\wp_dgst.c" />
<ClCompile Include="$(opensslDir)crypto\x509\by_dir.c" />
<ClCompile Include="$(opensslDir)crypto\x509\by_file.c" />
diff --git a/PCbuild/python.props b/PCbuild/python.props index 5f5ae75..03c4656 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -35,7 +35,7 @@ <sqlite3Dir>$(ExternalsDir)sqlite-3.14.2.0\</sqlite3Dir>
<bz2Dir>$(ExternalsDir)bzip2-1.0.6\</bz2Dir>
<bsddbDir>$(ExternalsDir)bsddb-4.7.25.0</bsddbDir>
- <opensslDir>$(ExternalsDir)openssl-1.0.2o\</opensslDir>
+ <opensslDir>$(ExternalsDir)openssl-1.0.2q\</opensslDir>
<opensslIncludeDir>$(opensslDir)include32</opensslIncludeDir>
<opensslIncludeDir Condition="'$(ArchName)' == 'amd64'">$(opensslDir)include64</opensslIncludeDir>
<nasmDir>$(ExternalsDir)\nasm-2.11.06\</nasmDir>
diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index e69ad94..893e8fd 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -70,7 +70,7 @@ <ClCompile>
<AdditionalOptions>/Zm200 %(AdditionalOptions)</AdditionalOptions>
<AdditionalIncludeDirectories>$(PySourcePath)Python;$(PySourcePath)Modules\zlib;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
- <PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
+ <PreprocessorDefinitions>_USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;_CRT_SECURE_NO_WARNINGS;MS_DLL_ID="$(SysWinVer)";%(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
<Link>
<AdditionalDependencies>ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies>
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py index 08353a9..ac61c78 100755 --- a/Parser/asdl_c.py +++ b/Parser/asdl_c.py @@ -830,7 +830,7 @@ static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena) static int obj2ast_int(PyObject* obj, int* out, PyArena* arena) { int i; - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!_PyAnyInt_Check(obj)) { PyObject *s = PyObject_Repr(obj); if (s == NULL) return 1; PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s", diff --git a/Parser/grammar.c b/Parser/grammar.c index fa1dd7c..fcd2219 100644 --- a/Parser/grammar.c +++ b/Parser/grammar.c @@ -32,6 +32,24 @@ newgrammar(int start) return g; } +void +freegrammar(grammar *g) +{ + int i; + for (i = 0; i < g->g_ndfas; i++) { + int j; + free(g->g_dfa[i].d_name); + for (j = 0; j < g->g_dfa[i].d_nstates; j++) + PyObject_FREE(g->g_dfa[i].d_state[j].s_arc); + PyObject_FREE(g->g_dfa[i].d_state); + } + PyObject_FREE(g->g_dfa); + for (i = 0; i < g->g_ll.ll_nlabels; i++) + free(g->g_ll.ll_label[i].lb_str); + PyObject_FREE(g->g_ll.ll_label); + PyObject_FREE(g); +} + dfa * adddfa(grammar *g, int type, char *name) { diff --git a/Parser/parsetok.c b/Parser/parsetok.c index 069cc6b..a5e9222 100644 --- a/Parser/parsetok.c +++ b/Parser/parsetok.c @@ -176,7 +176,7 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret, } else started = 1; - len = b - a; /* XXX this may compute NULL - NULL */ + len = (a != NULL && b != NULL) ? b - a : 0; str = (char *) PyObject_MALLOC(len + 1); if (str == NULL) { fprintf(stderr, "no mem for next token\n"); @@ -189,10 +189,12 @@ parsetok(struct tok_state *tok, grammar *g, int start, perrdetail *err_ret, #ifdef PY_PARSER_REQUIRES_FUTURE_KEYWORD #endif - if (a >= tok->line_start) + if (a != NULL && a >= tok->line_start) { col_offset = a - tok->line_start; - else + } + else { col_offset = -1; + } if ((err_ret->error = PyParser_AddToken(ps, (int)type, str, tok->lineno, col_offset, diff --git a/Parser/pgen.c b/Parser/pgen.c index f3031ae..b20d976 100644 --- a/Parser/pgen.c +++ b/Parser/pgen.c @@ -117,6 +117,17 @@ newnfagrammar(void) return gr; } +static void +freenfagrammar(nfagrammar *gr) +{ + int i; + for (i = 0; i < gr->gr_nnfas; i++) { + PyObject_FREE(gr->gr_nfa[i]->nf_state); + } + PyObject_FREE(gr->gr_nfa); + PyObject_FREE(gr); +} + static nfa * addnfa(nfagrammar *gr, char *name) { @@ -394,6 +405,7 @@ makedfa(nfagrammar *gr, nfa *nf, dfa *d) int istate, jstate, iarc, jarc, ibit; nfastate *st; nfaarc *ar; + int i, j; ss = newbitset(nbits); addclosure(ss, nf, nf->nf_start); @@ -488,7 +500,11 @@ makedfa(nfagrammar *gr, nfa *nf, dfa *d) convert(d, xx_nstates, xx_state); - /* XXX cleanup */ + for (i = 0; i < xx_nstates; i++) { + for (j = 0; j < xx_state[i].ss_narcs; j++) + delbitset(xx_state[i].ss_arc[j].sa_bitset); + PyObject_FREE(xx_state[i].ss_arc); + } PyObject_FREE(xx_state); } @@ -669,7 +685,7 @@ pgen(node *n) g = maketables(gr); translatelabels(g); addfirstsets(g); - PyObject_FREE(gr); + freenfagrammar(gr); return g; } diff --git a/Parser/pgenmain.c b/Parser/pgenmain.c index 88fa7f1..0b47295 100644 --- a/Parser/pgenmain.c +++ b/Parser/pgenmain.c @@ -67,6 +67,7 @@ main(int argc, char **argv) printf("Writing %s ...\n", graminit_h); printnonterminals(g, fp); fclose(fp); + freegrammar(g); Py_Exit(0); return 0; /* Make gcc -Wall happy */ } diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 4fdbad9..c6e61df 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1636,6 +1636,8 @@ tok_get(register struct tok_state *tok, char **p_start, char **p_end) "<> not supported in 3.x; use !=", tok->filename, tok->lineno, NULL, NULL)) { + tok->done = E_ERROR; + tok->cur = tok->inp; return ERRORTOKEN; } } @@ -1679,6 +1681,11 @@ int PyTokenizer_Get(struct tok_state *tok, char **p_start, char **p_end) { int result = tok_get(tok, p_start, p_end); + if (tok->fp && ferror(tok->fp)) { + clearerr(tok->fp); + result = ERRORTOKEN; + tok->done = E_IO; + } if (tok->decoding_erred) { result = ERRORTOKEN; tok->done = E_DECODE; diff --git a/Python/Python-ast.c b/Python/Python-ast.c index 4ac5cf5..2e7a1af 100644 --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -617,7 +617,7 @@ static int obj2ast_string(PyObject* obj, PyObject** out, PyArena* arena) static int obj2ast_int(PyObject* obj, int* out, PyArena* arena) { int i; - if (!PyInt_Check(obj) && !PyLong_Check(obj)) { + if (!_PyAnyInt_Check(obj)) { PyObject *s = PyObject_Repr(obj); if (s == NULL) return 1; PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s", diff --git a/Python/ast.c b/Python/ast.c index 318c0bb..9460325 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -2520,6 +2520,8 @@ alias_for_import_name(struct compiling *c, const node *n, int store) break; case STAR: str = PyString_InternFromString("*"); + if (!str) + return NULL; PyArena_AddPyObject(c->c_arena, str); return alias(str, NULL, c->c_arena); default: diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index 9ce3b27..4b819da 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -1780,7 +1780,7 @@ get_range_long_argument(PyObject *arg, const char *name) { PyObject *v; PyNumberMethods *nb; - if (PyInt_Check(arg) || PyLong_Check(arg)) { + if (_PyAnyInt_Check(arg)) { Py_INCREF(arg); return arg; } @@ -1795,7 +1795,7 @@ get_range_long_argument(PyObject *arg, const char *name) v = nb->nb_int(arg); if (v == NULL) return NULL; - if (PyInt_Check(v) || PyLong_Check(v)) + if (_PyAnyInt_Check(v)) return v; Py_DECREF(v); PyErr_SetString(PyExc_TypeError, @@ -2363,6 +2363,11 @@ builtin_sum(PyObject *self, PyObject *args) } /* Either overflowed or is not an int. Restore real objects and process normally */ result = PyInt_FromLong(i_result); + if (result == NULL) { + Py_DECREF(item); + Py_DECREF(iter); + return NULL; + } temp = PyNumber_Add(result, item); Py_DECREF(result); Py_DECREF(item); @@ -2401,6 +2406,11 @@ builtin_sum(PyObject *self, PyObject *args) continue; } result = PyFloat_FromDouble(f_result); + if (result == NULL) { + Py_DECREF(item); + Py_DECREF(iter); + return NULL; + } temp = PyNumber_Add(result, item); Py_DECREF(result); Py_DECREF(item); diff --git a/Python/ceval.c b/Python/ceval.c index b55b4d6..e1140a8 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -824,8 +824,12 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) PyObject *consts; #if defined(Py_DEBUG) || defined(LLTRACE) /* Make it easier to find out where we are with a debugger */ +#ifdef __GNUC__ + char *filename __attribute__((unused)); +#else char *filename; #endif +#endif /* Tuple access macros */ @@ -1288,7 +1292,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) FAST_DISPATCH(); } - + TARGET_NOARG(DUP_TOP) { v = TOP(); @@ -1835,7 +1839,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) } - + TARGET_WITH_IMPL_NOARG(SLICE, _slice) TARGET_WITH_IMPL_NOARG(SLICE_1, _slice) TARGET_WITH_IMPL_NOARG(SLICE_2, _slice) @@ -1860,7 +1864,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) break; } - + TARGET_WITH_IMPL_NOARG(STORE_SLICE, _store_slice) TARGET_WITH_IMPL_NOARG(STORE_SLICE_1, _store_slice) TARGET_WITH_IMPL_NOARG(STORE_SLICE_2, _store_slice) @@ -4750,8 +4754,7 @@ _PyEval_SliceIndexNotNone(PyObject *v, Py_ssize_t *pi) #undef ISINDEX -#define ISINDEX(x) ((x) == NULL || \ - PyInt_Check(x) || PyLong_Check(x) || PyIndex_Check(x)) +#define ISINDEX(x) ((x) == NULL || _PyAnyInt_Check(x) || PyIndex_Check(x)) static PyObject * apply_slice(PyObject *u, PyObject *v, PyObject *w) /* return u[v:w] */ @@ -5244,7 +5247,7 @@ getarray(long a[256]) Py_DECREF(l); return NULL; } - PyList_SetItem(l, i, x); + PyList_SET_ITEM(l, i, x); } for (i = 0; i < 256; i++) a[i] = 0; @@ -5266,7 +5269,7 @@ _Py_GetDXProfile(PyObject *self, PyObject *args) Py_DECREF(l); return NULL; } - PyList_SetItem(l, i, x); + PyList_SET_ITEM(l, i, x); } return l; #endif diff --git a/Python/getargs.c b/Python/getargs.c index 32ff6fc..cc1ddde 100644 --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1780,7 +1780,7 @@ skipitem(const char **p_format, va_list *p_va, int flags) else (void) va_arg(*p_va, int *); format++; - } else if ((c == 's' || c == 'z') && *format == '*') { + } else if ((c == 's' || c == 'z' || c == 'w') && *format == '*') { format++; } break; diff --git a/Python/getcopyright.c b/Python/getcopyright.c index 1b69012..0ef16d0 100644 --- a/Python/getcopyright.c +++ b/Python/getcopyright.c @@ -4,7 +4,7 @@ static char cprt[] = "\ -Copyright (c) 2001-2018 Python Software Foundation.\n\ +Copyright (c) 2001-2019 Python Software Foundation.\n\ All Rights Reserved.\n\ \n\ Copyright (c) 2000 BeOpen.com.\n\ diff --git a/Python/import.c b/Python/import.c index 1d74faf..ccbd949 100644 --- a/Python/import.c +++ b/Python/import.c @@ -447,7 +447,9 @@ PyImport_Cleanup(void) dict = PyModule_GetDict(value); if (Py_VerboseFlag) PySys_WriteStderr("# clear __builtin__._\n"); - PyDict_SetItemString(dict, "_", Py_None); + if (PyDict_SetItemString(dict, "_", Py_None) < 0) { + PyErr_Clear(); + } } value = PyDict_GetItemString(modules, "sys"); if (value != NULL && PyModule_Check(value)) { @@ -457,7 +459,9 @@ PyImport_Cleanup(void) for (p = sys_deletes; *p != NULL; p++) { if (Py_VerboseFlag) PySys_WriteStderr("# clear sys.%s\n", *p); - PyDict_SetItemString(dict, *p, Py_None); + if (PyDict_SetItemString(dict, *p, Py_None) < 0) { + PyErr_Clear(); + } } for (p = sys_files; *p != NULL; p+=2) { if (Py_VerboseFlag) @@ -465,7 +469,9 @@ PyImport_Cleanup(void) v = PyDict_GetItemString(dict, *(p+1)); if (v == NULL) v = Py_None; - PyDict_SetItemString(dict, *p, v); + if (PyDict_SetItemString(dict, *p, v) < 0) { + PyErr_Clear(); + } } } @@ -475,7 +481,9 @@ PyImport_Cleanup(void) if (Py_VerboseFlag) PySys_WriteStderr("# cleanup __main__\n"); _PyModule_Clear(value); - PyDict_SetItemString(modules, "__main__", Py_None); + if (PyDict_SetItemString(modules, "__main__", Py_None) < 0) { + PyErr_Clear(); + } } /* The special treatment of __builtin__ here is because even @@ -510,10 +518,15 @@ PyImport_Cleanup(void) PySys_WriteStderr( "# cleanup[1] %s\n", name); _PyModule_Clear(value); - PyDict_SetItem(modules, key, Py_None); + if (PyDict_SetItem(modules, key, Py_None) < 0) { + PyErr_Clear(); + } ndone++; } } + if (PyErr_Occurred()) { + PyErr_Clear(); + } } while (ndone > 0); /* Next, delete all modules (still skipping __builtin__ and sys) */ @@ -528,7 +541,12 @@ PyImport_Cleanup(void) if (Py_VerboseFlag) PySys_WriteStderr("# cleanup[2] %s\n", name); _PyModule_Clear(value); - PyDict_SetItem(modules, key, Py_None); + if (PyDict_SetItem(modules, key, Py_None) < 0) { + PyErr_Clear(); + } + } + if (PyErr_Occurred()) { + PyErr_Clear(); } } @@ -538,14 +556,18 @@ PyImport_Cleanup(void) if (Py_VerboseFlag) PySys_WriteStderr("# cleanup sys\n"); _PyModule_Clear(value); - PyDict_SetItemString(modules, "sys", Py_None); + if (PyDict_SetItemString(modules, "sys", Py_None) < 0) { + PyErr_Clear(); + } } value = PyDict_GetItemString(modules, "__builtin__"); if (value != NULL && PyModule_Check(value)) { if (Py_VerboseFlag) PySys_WriteStderr("# cleanup __builtin__\n"); _PyModule_Clear(value); - PyDict_SetItemString(modules, "__builtin__", Py_None); + if (PyDict_SetItemString(modules, "__builtin__", Py_None) < 0) { + PyErr_Clear(); + } } /* Finally, clear and delete the modules directory */ @@ -679,7 +701,7 @@ remove_module(const char *name) if (PyDict_GetItemString(modules, name) == NULL) return; if (PyDict_DelItemString(modules, name) < 0) - Py_FatalError("import: deleting existing key in" + Py_FatalError("import: deleting existing key in " "sys.modules failed"); } diff --git a/Python/pystrtod.c b/Python/pystrtod.c index ae6ab9c..29d7996 100644 --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -1004,8 +1004,6 @@ format_float_short(double d, char format_code, else { /* shouldn't get here: Gay's code should always return something starting with a digit, an 'I', or 'N' */ - strncpy(p, "ERR", 3); - p += 3; assert(0); } goto exit; diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 44fe13d..2c9f55f 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1136,7 +1136,7 @@ handle_system_exit(void) /* If we failed to dig out the 'code' attribute, just let the else clause below print the error. */ } - if (PyInt_Check(value) || PyLong_Check(value)) + if (_PyAnyInt_Check(value)) exitcode = (int)PyInt_AsLong(value); else { PyObject *sys_stderr = PySys_GetObject("stderr"); @@ -1590,7 +1590,7 @@ err_input(perrdetail *err) errtype = PyExc_SyntaxError; switch (err->error) { case E_ERROR: - return; + goto cleanup; case E_SYNTAX: errtype = PyExc_IndentationError; if (err->expected == INDENT) @@ -1654,6 +1654,9 @@ err_input(perrdetail *err) Py_XDECREF(tb); break; } + case E_IO: + msg = "I/O error while reading"; + break; case E_LINECONT: msg = "unexpected character after line continuation character"; break; diff --git a/Python/symtable.c b/Python/symtable.c index 21790b1..23aeaaa 100644 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -199,8 +199,10 @@ symtable_new(void) struct symtable *st; st = (struct symtable *)PyMem_Malloc(sizeof(struct symtable)); - if (st == NULL) + if (st == NULL) { + PyErr_NoMemory(); return NULL; + } st->st_filename = NULL; st->st_symbols = NULL; diff --git a/Python/sysmodule.c b/Python/sysmodule.c index b153ef6..fdb7af2 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1555,7 +1555,7 @@ makepathobject(char *path, int delim) Py_DECREF(v); return NULL; } - PyList_SetItem(v, i, w); + PyList_SET_ITEM(v, i, w); if (*p == '\0') break; path = p+1; diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index 79c66d4..6d4b3b3 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -156,6 +156,28 @@ PyThread__init_thread(void) * Thread support. */ +/* bpo-33015: pythread_callback struct and pythread_wrapper() cast + "void func(void *)" to "void* func(void *)": always return NULL. + + PyThread_start_new_thread() uses "void func(void *)" type, whereas + pthread_create() requires a void* return value. */ +typedef struct { + void (*func) (void *); + void *arg; +} pythread_callback; + +static void * +pythread_wrapper(void *arg) +{ + /* copy func and func_arg and free the temporary structure */ + pythread_callback *callback = arg; + void (*func)(void *) = callback->func; + void *func_arg = callback->arg; + free(arg); + + func(func_arg); + return NULL; +} long PyThread_start_new_thread(void (*func)(void *), void *arg) @@ -191,21 +213,31 @@ PyThread_start_new_thread(void (*func)(void *), void *arg) pthread_attr_setscope(&attrs, PTHREAD_SCOPE_SYSTEM); #endif + pythread_callback *callback = malloc(sizeof(pythread_callback)); + + if (callback == NULL) { + return -1; + } + + callback->func = func; + callback->arg = arg; + status = pthread_create(&th, #if defined(THREAD_STACK_SIZE) || defined(PTHREAD_SYSTEM_SCHED_SUPPORTED) &attrs, #else (pthread_attr_t*)NULL, #endif - (void* (*)(void *))func, - (void *)arg - ); + pythread_wrapper, callback); #if defined(THREAD_STACK_SIZE) || defined(PTHREAD_SYSTEM_SCHED_SUPPORTED) pthread_attr_destroy(&attrs); #endif - if (status != 0) + + if (status != 0) { + free(callback); return -1; + } pthread_detach(th); @@ -1,9 +1,9 @@ -This is Python version 2.7.15 +This is Python version 2.7.16 ============================= Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation. All rights -reserved. +2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation. All +rights reserved. Copyright (c) 2000 BeOpen.com. All rights reserved. diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index e218a31..9def56e 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -241,12 +241,13 @@ class PyObjectPtr(object): def safe_tp_name(self): try: - return self.type().field('tp_name').string() - except NullPyObjectPtr: - # NULL tp_name? - return 'unknown' - except RuntimeError: - # Can't even read the object at all? + ob_type = self.type() + tp_name = ob_type.field('tp_name') + return tp_name.string() + # NullPyObjectPtr: NULL tp_name? + # RuntimeError: Can't even read the object at all? + # UnicodeDecodeError: Failed to decode tp_name bytestring + except (NullPyObjectPtr, RuntimeError, UnicodeDecodeError): return 'unknown' def proxyval(self, visited): @@ -320,7 +321,9 @@ class PyObjectPtr(object): try: tp_name = t.field('tp_name').string() tp_flags = int(t.field('tp_flags')) - except RuntimeError: + # RuntimeError: NULL pointers + # UnicodeDecodeError: string() fails to decode the bytestring + except (RuntimeError, UnicodeDecodeError): # Handle any kind of error e.g. NULL ptrs by simply using the base # class return cls @@ -336,6 +339,7 @@ class PyObjectPtr(object): 'set' : PySetObjectPtr, 'frozenset' : PySetObjectPtr, 'builtin_function_or_method' : PyCFunctionObjectPtr, + 'method-wrapper': wrapperobject, } if tp_name in name_map: return name_map[tp_name] @@ -602,7 +606,10 @@ class PyCFunctionObjectPtr(PyObjectPtr): def proxyval(self, visited): m_ml = self.field('m_ml') # m_ml is a (PyMethodDef*) - ml_name = m_ml['ml_name'].string() + try: + ml_name = m_ml['ml_name'].string() + except UnicodeDecodeError: + ml_name = '<ml_name:UnicodeDecodeError>' pyop_m_self = self.pyop_field('m_self') if pyop_m_self.is_null(): @@ -915,35 +922,50 @@ class PyFrameObjectPtr(PyObjectPtr): if long(f_trace) != 0: # we have a non-NULL f_trace: return self.f_lineno - else: - #try: + + try: return self.co.addr2line(self.f_lasti) - #except ValueError: - # return self.f_lineno + except Exception: + # bpo-34989: addr2line() is a complex function, it can fail in many + # ways. For example, it fails with a TypeError on "FakeRepr" if + # gdb fails to load debug symbols. Use a catch-all "except + # Exception" to make the whole function safe. The caller has to + # handle None anyway for optimized Python. + return None def current_line(self): '''Get the text of the current source line as a string, with a trailing newline character''' if self.is_optimized_out(): return '(frame information optimized out)' + + lineno = self.current_line_num() + if lineno is None: + return '(failed to get frame line number)' + filename = self.filename() try: - f = open(filename, 'r') + with open(filename, 'r') as fp: + lines = fp.readlines() except IOError: return None - with f: - all_lines = f.readlines() - # Convert from 1-based current_line_num to 0-based list offset: - return all_lines[self.current_line_num()-1] + + try: + # Convert from 1-based current_line_num to 0-based list offset + return lines[lineno - 1] + except IndexError: + return None def write_repr(self, out, visited): if self.is_optimized_out(): out.write('(frame information optimized out)') return - out.write('Frame 0x%x, for file %s, line %i, in %s (' + lineno = self.current_line_num() + lineno = str(lineno) if lineno is not None else "?" + out.write('Frame 0x%x, for file %s, line %s, in %s (' % (self.as_address(), self.co_filename.proxyval(visited), - self.current_line_num(), + lineno, self.co_name.proxyval(visited))) first = True for pyop_name, pyop_value in self.iter_locals(): @@ -962,9 +984,11 @@ class PyFrameObjectPtr(PyObjectPtr): sys.stdout.write(' (frame information optimized out)\n') return visited = set() - sys.stdout.write(' File "%s", line %i, in %s\n' + lineno = self.current_line_num() + lineno = str(lineno) if lineno is not None else "?" + sys.stdout.write(' File "%s", line %s, in %s\n' % (self.co_filename.proxyval(visited), - self.current_line_num(), + lineno, self.co_name.proxyval(visited))) class PySetObjectPtr(PyObjectPtr): @@ -1131,7 +1155,9 @@ class PyUnicodeObjectPtr(PyObjectPtr): # Convert the int code points to unicode characters, and generate a # local unicode instance. # This splits surrogate pairs if sizeof(Py_UNICODE) is 2 here (in gdb). - result = u''.join([_unichr(ucs) for ucs in Py_UNICODEs]) + result = u''.join([ + (_unichr(ucs) if ucs <= 0x10ffff else '\ufffd') + for ucs in Py_UNICODEs]) return result def write_repr(self, out, visited): @@ -1144,6 +1170,41 @@ class PyUnicodeObjectPtr(PyObjectPtr): out.write(val.lstrip('u')) +class wrapperobject(PyObjectPtr): + _typename = 'wrapperobject' + + def safe_name(self): + try: + name = self.field('descr')['d_base']['name'].string() + return repr(name) + except (NullPyObjectPtr, RuntimeError, UnicodeDecodeError): + return '<unknown name>' + + def safe_tp_name(self): + try: + return self.field('self')['ob_type']['tp_name'].string() + except (NullPyObjectPtr, RuntimeError, UnicodeDecodeError): + return '<unknown tp_name>' + + def safe_self_addresss(self): + try: + address = long(self.field('self')) + return '%#x' % address + except (NullPyObjectPtr, RuntimeError): + return '<failed to get self address>' + + def proxyval(self, visited): + name = self.safe_name() + tp_name = self.safe_tp_name() + self_address = self.safe_self_addresss() + return ("<method-wrapper %s of %s object at %s>" + % (name, tp_name, self_address)) + + def write_repr(self, out, visited): + proxy = self.proxyval(visited) + out.write(proxy) + + def int_from_int(gdbval): return int(str(gdbval)) @@ -1176,11 +1237,13 @@ class PyObjectPtrPrinter: def pretty_printer_lookup(gdbval): type = gdbval.type.unqualified() - if type.code == gdb.TYPE_CODE_PTR: - type = type.target().unqualified() - t = str(type) - if t in ("PyObject", "PyFrameObject"): - return PyObjectPtrPrinter(gdbval) + if type.code != gdb.TYPE_CODE_PTR: + return None + + type = type.target().unqualified() + t = str(type) + if t in ("PyObject", "PyFrameObject", "PyUnicodeObject", "wrapperobject"): + return PyObjectPtrPrinter(gdbval) """ During development, I've been manually invoking the code in this way: @@ -1202,7 +1265,7 @@ that this python file is installed to the same path as the library (or its /usr/lib/debug/usr/lib/libpython2.6.so.1.0.debug-gdb.py """ def register (obj): - if obj == None: + if obj is None: obj = gdb # Wire up the pretty-printer @@ -1304,23 +1367,43 @@ class Frame(object): ''' if self.is_waiting_for_gil(): return 'Waiting for the GIL' - elif self.is_gc_collect(): + + if self.is_gc_collect(): return 'Garbage-collecting' - else: - # Detect invocations of PyCFunction instances: - older = self.older() - if older and older._gdbframe.name() == 'PyCFunction_Call': - # Within that frame: - # "func" is the local containing the PyObject* of the - # PyCFunctionObject instance - # "f" is the same value, but cast to (PyCFunctionObject*) - # "self" is the (PyObject*) of the 'self' - try: - # Use the prettyprinter for the func: - func = older._gdbframe.read_var('func') - return str(func) - except RuntimeError: - return 'PyCFunction invocation (unable to read "func")' + + # Detect invocations of PyCFunction instances: + frame = self._gdbframe + caller = frame.name() + if not caller: + return False + + if caller == 'PyCFunction_Call': + arg_name = 'func' + # Within that frame: + # "func" is the local containing the PyObject* of the + # PyCFunctionObject instance + # "f" is the same value, but cast to (PyCFunctionObject*) + # "self" is the (PyObject*) of the 'self' + try: + # Use the prettyprinter for the func: + func = frame.read_var(arg_name) + return str(func) + except ValueError: + return ('PyCFunction invocation (unable to read %s: ' + 'missing debuginfos?)' % arg_name) + except RuntimeError: + return 'PyCFunction invocation (unable to read %s)' % arg_name + + if caller == 'wrapper_call': + arg_name = 'wp' + try: + func = frame.read_var(arg_name) + return str(func) + except ValueError: + return ('<wrapper_call invocation (unable to read %s: ' + 'missing debuginfos?)>' % arg_name) + except RuntimeError: + return '<wrapper_call invocation (unable to read %s)>' % arg_name # This frame isn't worth reporting: return False @@ -1368,7 +1451,11 @@ class Frame(object): def get_selected_python_frame(cls): '''Try to obtain the Frame for the python-related code in the selected frame, or None''' - frame = cls.get_selected_frame() + try: + frame = cls.get_selected_frame() + except gdb.error: + # No frame: Python didn't start yet + return None while frame: if frame.is_python_frame(): @@ -1476,6 +1563,9 @@ class PyList(gdb.Command): filename = pyop.filename() lineno = pyop.current_line_num() + if lineno is None: + print('Unable to read python frame line number') + return if start is None: start = lineno - 5 @@ -1509,6 +1599,10 @@ PyList() def move_in_stack(move_up): '''Move up or down the stack (for the py-up/py-down command)''' frame = Frame.get_selected_python_frame() + if not frame: + print('Unable to locate python frame') + return + while frame: if move_up: iter_frame = frame.older() @@ -1571,6 +1665,10 @@ class PyBacktraceFull(gdb.Command): def invoke(self, args, from_tty): frame = Frame.get_selected_python_frame() + if not frame: + print('Unable to locate python frame') + return + while frame: if frame.is_python_frame(): frame.print_summary() @@ -1588,8 +1686,12 @@ class PyBacktrace(gdb.Command): def invoke(self, args, from_tty): - sys.stdout.write('Traceback (most recent call first):\n') frame = Frame.get_selected_python_frame() + if not frame: + print('Unable to locate python frame') + return + + sys.stdout.write('Traceback (most recent call first):\n') while frame: if frame.is_python_frame(): frame.print_traceback() diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py index 65ce5c1..da4a1c3 100644 --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -917,7 +917,7 @@ def generate_license(): shutil.copyfileobj(open(os.path.join(srcdir, "LICENSE")), out) shutil.copyfileobj(open("crtlicense.txt"), out) for name, pat, file in (("bzip2","bzip2-*", "LICENSE"), - ("Berkeley DB", "db-*", "LICENSE"), + ("Berkeley DB", "bsddb-*", "LICENSE"), ("openssl", "openssl-*", "LICENSE"), ("Tcl", "tcl-8*", "license.terms"), ("Tk", "tk-8*", "license.terms"), diff --git a/Tools/msi/uuids.py b/Tools/msi/uuids.py index 6c27483..b6c1c06 100644 --- a/Tools/msi/uuids.py +++ b/Tools/msi/uuids.py @@ -80,4 +80,14 @@ product_codes = { '2.7.14150':'{0398A685-FD8D-46B3-9816-C47319B0CF5E}', # 2.7.14 '2.7.15121':'{B4B95EEC-7227-4F69-A918-043AF75653AC}', # 2.7.15rc1 '2.7.15150':'{16CD92A4-0152-4CB7-8FD6-9788D3363616}', # 2.7.15 + '2.7.16121':'{06C30F70-A78F-41ED-983C-22F6AD42FC1E}', # 2.7.16rc1 + '2.7.16150':'{DCD5B320-89D9-4C7C-9E8B-84496588744D}', # 2.7.16 + '2.7.17121':'{90B71DF4-FCF6-4CB1-B843-0C764DBAC649}', # 2.7.17rc1 + '2.7.17150':'{9255D53C-6C21-4664-AAF3-6EAC50F867D9}', # 2.7.17 + '2.7.18121':'{51145CAA-9FFF-4AF5-86CF-1AA96A693DA7}', # 2.7.18rc1 + '2.7.18150':'{A5F504DF-2ED9-4A2D-A2F3-9D2750DD42D5}', # 2.7.18 + '2.7.19121':'{771232C8-7A9F-443B-BD87-848075733F0D}', # 2.7.19rc1 + '2.7.19150':'{35C139B3-F743-42AE-AA69-CCAEFC692FF9}', # 2.7.19 + '2.7.20121':'{2BA2FC48-5429-462A-93CD-2004D094CF1E}', # 2.7.20rc1 + '2.7.20150':'{5C951F3A-4522-4AD0-9E7C-A82D06E99BC8}', # 2.7.20 } diff --git a/Tools/scripts/diff.py b/Tools/scripts/diff.py index 513e2a7..c4c2e10 100755 --- a/Tools/scripts/diff.py +++ b/Tools/scripts/diff.py @@ -32,8 +32,10 @@ def main(): fromdate = time.ctime(os.stat(fromfile).st_mtime) todate = time.ctime(os.stat(tofile).st_mtime) - fromlines = open(fromfile, 'U').readlines() - tolines = open(tofile, 'U').readlines() + with open(fromfile, 'U') as f: + fromlines = f.readlines() + with open(tofile, 'U') as f: + tolines = f.readlines() if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n) diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py index b69d540..58aa168 100755 --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -43,20 +43,6 @@ def status(message, modal=False, info=None): return decorated_fxn -def mq_patches_applied(): - """Check if there are any applied MQ patches.""" - cmd = 'hg qapplied' - st = subprocess.Popen(cmd.split(), - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - try: - bstdout, _ = st.communicate() - return st.returncode == 0 and bstdout - finally: - st.stdout.close() - st.stderr.close() - - def get_git_branch(): """Get the symbolic name for the current git branch""" cmd = "git rev-parse --abbrev-ref HEAD".split() @@ -101,19 +87,8 @@ def get_base_branch(): @status("Getting the list of files that have been added/changed", info=lambda x: n_files_str(len(x))) def changed_files(base_branch=None): - """Get the list of changed or added files from Mercurial or git.""" - if os.path.isdir(os.path.join(SRCDIR, '.hg')): - if base_branch is not None: - sys.exit('need a git checkout to check PR status') - cmd = 'hg status --added --modified --no-status' - if mq_patches_applied(): - cmd += ' --rev qparent' - st = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE) - try: - filenames = [x.decode().rstrip() for x in st.stdout] - finally: - st.stdout.close() - elif os.path.exists(os.path.join(SRCDIR, '.git')): + """Get the list of changed or added files from git.""" + if os.path.exists(os.path.join(SRCDIR, '.git')): # We just use an existence check here as: # directory = normal git checkout/clone # file = git worktree directory @@ -138,7 +113,7 @@ def changed_files(base_branch=None): finally: st.stdout.close() else: - sys.exit('need a Mercurial or git checkout to get modified files') + sys.exit('need a git checkout to get modified files') filenames2 = [] for filename in filenames: @@ -221,10 +196,11 @@ def credit_given(file_paths): return os.path.join('Misc', 'ACKS') in file_paths -@status("Misc/NEWS updated", modal=True) +@status("Misc/NEWS.d updated with `blurb`", modal=True) def reported_news(file_paths): - """Check if Misc/NEWS has been changed.""" - return os.path.join('Misc', 'NEWS') in file_paths + """Check if Misc/NEWS.d has been changed.""" + return any(p.startswith(os.path.join('Misc', 'NEWS.d', 'next')) + for p in file_paths) def main(): @@ -234,8 +210,7 @@ def main(): c_files = [fn for fn in file_paths if fn.endswith(('.c', '.h'))] doc_files = [fn for fn in file_paths if fn.startswith('Doc') and fn.endswith(('.rst', '.inc'))] - misc_files = {os.path.join('Misc', 'ACKS'), os.path.join('Misc', 'NEWS')}\ - & set(file_paths) + misc_files = {p for p in file_paths if p.startswith('Misc')} # PEP 8 whitespace rules enforcement. normalize_whitespace(python_files) # C rules enforcement. diff --git a/Tools/scripts/texi2html.py b/Tools/scripts/texi2html.py index 57db166..fbf32d6 100755 --- a/Tools/scripts/texi2html.py +++ b/Tools/scripts/texi2html.py @@ -1808,7 +1808,7 @@ class HTMLHelp: print>>fp, '<!-- This file defines the table of contents -->' print>>fp, '<HTML>' print>>fp, '<HEAD>' - print>>fp, ('<meta name="GENERATOR"' + print>>fp, ('<meta name="GENERATOR" ' 'content="Microsoft® HTML Help Workshop 4.1">') print>>fp, '<!-- Sitemap 1.0 -->' print>>fp, '</HEAD>' @@ -1833,7 +1833,7 @@ class HTMLHelp: print>>fp, '<!-- This file defines the index -->' print>>fp, '<HTML>' print>>fp, '<HEAD>' - print>>fp, ('<meta name="GENERATOR"' + print>>fp, ('<meta name="GENERATOR" ' 'content="Microsoft® HTML Help Workshop 4.1">') print>>fp, '<!-- Sitemap 1.0 -->' print>>fp, '</HEAD>' diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py index ffc57f0..78b068a 100755 --- a/Tools/ssl/multissltests.py +++ b/Tools/ssl/multissltests.py @@ -41,15 +41,14 @@ import tarfile log = logging.getLogger("multissl") OPENSSL_OLD_VERSIONS = [ - "0.9.8zc", - "0.9.8zh", "1.0.1u", + "1.0.2o", ] OPENSSL_RECENT_VERSIONS = [ - "1.0.2", - "1.0.2l", - "1.1.0f", + "1.0.2p", + "1.1.0i", + "1.1.1", ] LIBRESSL_OLD_VERSIONS = [ @@ -59,13 +58,15 @@ LIBRESSL_OLD_VERSIONS = [ LIBRESSL_RECENT_VERSIONS = [ "2.5.5", - "2.6.4", - "2.7.1", + "2.6.5", + "2.7.4", ] # store files in ../multissl -HERE = os.path.abspath(os.getcwd()) -MULTISSL_DIR = os.path.abspath(os.path.join(HERE, '..', 'multissl')) +HERE = os.path.dirname(os.path.abspath(__file__)) +PYTHONROOT = os.path.abspath(os.path.join(HERE, '..', '..')) +MULTISSL_DIR = os.path.abspath(os.path.join(PYTHONROOT, '..', 'multissl')) + parser = argparse.ArgumentParser( prog='multissl', @@ -77,7 +78,7 @@ parser = argparse.ArgumentParser( parser.add_argument( '--debug', action='store_true', - help="Enable debug mode", + help="Enable debug logging", ) parser.add_argument( '--disable-ancient', @@ -120,32 +121,49 @@ parser.add_argument( help="Disable network tests." ) parser.add_argument( - '--compile-only', - action='store_true', - help="Don't run tests, only compile _ssl.c and _hashopenssl.c." + '--steps', + choices=['library', 'modules', 'tests'], + default='tests', + help=( + "Which steps to perform. 'library' downloads and compiles OpenSSL " + "or LibreSSL. 'module' also compiles Python modules. 'tests' builds " + "all and runs the test suite." + ) ) +parser.add_argument( + '--force', + action='store_true', + dest='force', + help="Force build and installation." +) +parser.add_argument( + '--keep-sources', + action='store_true', + dest='keep_sources', + help="Keep original sources for debugging." +) class AbstractBuilder(object): library = None url_template = None src_template = None build_template = None + install_target = 'install' module_files = ("Modules/_ssl.c", "Modules/_hashopenssl.c") module_libs = ("_ssl", "_hashlib") - def __init__(self, version, compile_args=(), - basedir=MULTISSL_DIR): + def __init__(self, version, args): self.version = version - self.compile_args = compile_args + self.args = args # installation directory self.install_dir = os.path.join( - os.path.join(basedir, self.library.lower()), version + os.path.join(args.base_directory, self.library.lower()), version ) # source file - self.src_dir = os.path.join(basedir, 'src') + self.src_dir = os.path.join(args.base_directory, 'src') self.src_file = os.path.join( self.src_dir, self.src_template.format(version)) # build directory (removed after install) @@ -253,20 +271,29 @@ class AbstractBuilder(object): """Now build openssl""" log.info("Running build in {}".format(self.build_dir)) cwd = self.build_dir - cmd = ["./config", "shared", "--prefix={}".format(self.install_dir)] - cmd.extend(self.compile_args) + cmd = [ + "./config", + "shared", "--debug", + "--prefix={}".format(self.install_dir) + ] + env = os.environ.copy() + # set rpath + env["LD_RUN_PATH"] = self.lib_dir self._subprocess_call(cmd, cwd=cwd) # Old OpenSSL versions do not support parallel builds. self._subprocess_call(["make", "-j1"], cwd=cwd) - def _make_install(self, remove=True): - self._subprocess_call(["make", "-j1", "install"], cwd=self.build_dir) - if remove: + def _make_install(self): + self._subprocess_call( + ["make", "-j1", self.install_target], + cwd=self.build_dir + ) + if not self.args.keep_sources: shutil.rmtree(self.build_dir) def install(self): log.info(self.openssl_cli) - if not self.has_openssl: + if not self.has_openssl or self.args.force: if not self.has_src: self._download_src() else: @@ -332,6 +359,8 @@ class BuildOpenSSL(AbstractBuilder): url_template = "https://www.openssl.org/source/openssl-{}.tar.gz" src_template = "openssl-{}.tar.gz" build_template = "openssl-{}" + # only install software, skip docs + install_target = 'install_sw' class BuildLibreSSL(AbstractBuilder): @@ -370,57 +399,64 @@ def main(): start = datetime.now() - for name in ['python', 'setup.py', 'Modules/_ssl.c']: - if not os.path.isfile(name): + if args.steps in {'modules', 'tests'}: + for name in ['setup.py', 'Modules/_ssl.c']: + if not os.path.isfile(os.path.join(PYTHONROOT, name)): + parser.error( + "Must be executed with ./python from CPython build dir" + ) + if not os.path.samefile('python', sys.executable): parser.error( - "Must be executed from CPython build dir" - ) - if not os.path.samefile('python', sys.executable): - parser.error( "Must be executed with ./python from CPython build dir" ) - # check for configure and run make - configure_make() + # check for configure and run make + configure_make() # download and register builder builds = [] for version in args.openssl: - build = BuildOpenSSL(version) + build = BuildOpenSSL( + version, + args + ) build.install() builds.append(build) for version in args.libressl: - build = BuildLibreSSL(version) + build = BuildLibreSSL( + version, + args + ) build.install() builds.append(build) - for build in builds: - try: - build.recompile_pymods() - build.check_pyssl() - if not args.compile_only: - build.run_python_tests( - tests=args.tests, - network=args.network, - ) - except Exception as e: - log.exception("%s failed", build) - print("{} failed: {}".format(build, e), file=sys.stderr) - sys.exit(2) - - print("\n{} finished in {}".format( - "Tests" if not args.compile_only else "Builds", - datetime.now() - start - )) + if args.steps in {'modules', 'tests'}: + for build in builds: + try: + build.recompile_pymods() + build.check_pyssl() + if args.steps == 'tests': + build.run_python_tests( + tests=args.tests, + network=args.network, + ) + except Exception as e: + log.exception("%s failed", build) + print("{} failed: {}".format(build, e), file=sys.stderr) + sys.exit(2) + + log.info("\n{} finished in {}".format( + args.steps.capitalize(), + datetime.now() - start + )) print('Python: ', sys.version) - if args.compile_only: - print('Build only') - elif args.tests: - print('Executed Tests:', ' '.join(args.tests)) - else: - print('Executed all SSL tests.') + if args.steps == 'tests': + if args.tests: + print('Executed Tests:', ' '.join(args.tests)) + else: + print('Executed all SSL tests.') print('OpenSSL / LibreSSL versions:') for build in builds: @@ -1,6 +1,6 @@ -# generated automatically by aclocal 1.15 -*- Autoconf -*- +# generated automatically by aclocal 1.15.1 -*- Autoconf -*- -# Copyright (C) 1996-2014 Free Software Foundation, Inc. +# Copyright (C) 1996-2017 Free Software Foundation, Inc. # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -3040,7 +3040,7 @@ if test "${enable_universalsdk+set}" = set; then : enableval=$enable_universalsdk; case $enableval in yes) - # Locate the best usable SDK, see Mac/README.txt for more + # Locate the best usable SDK, see Mac/README for more # information enableval="`/usr/bin/xcodebuild -version -sdk macosx Path 2>/dev/null`" if ! ( echo $enableval | grep -E '\.sdk' 1>/dev/null ) @@ -5791,9 +5791,9 @@ HAS_GIT=no-repository fi if test $HAS_GIT = found then - GITVERSION="git -C \$(srcdir) rev-parse --short HEAD" - GITTAG="git -C \$(srcdir) describe --all --always --dirty" - GITBRANCH="git -C \$(srcdir) name-rev --name-only HEAD" + GITVERSION="git --git-dir \$(srcdir)/.git rev-parse --short HEAD" + GITTAG="git --git-dir \$(srcdir)/.git describe --all --always --dirty" + GITBRANCH="git --git-dir \$(srcdir)/.git name-rev --name-only HEAD" else GITVERSION="" GITTAG="" @@ -6481,6 +6481,13 @@ if test "$Py_LTO" = 'true' ; then esac ;; esac + + if test "$ac_cv_prog_cc_g" = "yes" + then + # bpo-30345: Add -g to LDFLAGS when compiling with LTO + # to get debug symbols. + LTOFLAGS="$LTOFLAGS -g" + fi fi @@ -10586,7 +10593,7 @@ for ac_func in alarm setitimer getitimer bind_textdomain_codeset chown \ gai_strerror getgroups getlogin getloadavg getpeername getpgid getpid \ getentropy \ getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \ - initgroups kill killpg lchmod lchown lstat mkfifo mknod mktime mmap \ + initgroups kill killpg lchown lstat mkfifo mknod mktime mmap \ mremap nice pathconf pause plock poll pthread_init \ putenv readlink realpath \ select sem_open sem_timedwait sem_getvalue sem_unlink setegid seteuid \ @@ -10609,6 +10616,17 @@ fi done +# Force lchmod off for Linux. Linux disallows changing the mode of symbolic +# links. Some libc implementations have a stub lchmod implementation that always +# returns an error. +if test "$MACHDEP" != linux; then + ac_fn_c_check_func "$LINENO" "lchmod" "ac_cv_func_lchmod" +if test "x$ac_cv_func_lchmod" = xyes; then : + +fi + +fi + # For some functions, having a definition is not sufficient, since # we want to take their address. { $as_echo "$as_me:${as_lineno-$LINENO}: checking for chroot" >&5 @@ -14349,77 +14367,6 @@ $as_echo "#define HAVE_BROKEN_POLL 1" >>confdefs.h fi -# Before we can test tzset, we need to check if struct tm has a tm_zone -# (which is not required by ISO C or UNIX spec) and/or if we support -# tzname[] -ac_fn_c_check_member "$LINENO" "struct tm" "tm_zone" "ac_cv_member_struct_tm_tm_zone" "#include <sys/types.h> -#include <$ac_cv_struct_tm> - -" -if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then : - -cat >>confdefs.h <<_ACEOF -#define HAVE_STRUCT_TM_TM_ZONE 1 -_ACEOF - - -fi - -if test "$ac_cv_member_struct_tm_tm_zone" = yes; then - -$as_echo "#define HAVE_TM_ZONE 1" >>confdefs.h - -else - ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h> -" -if test "x$ac_cv_have_decl_tzname" = xyes; then : - ac_have_decl=1 -else - ac_have_decl=0 -fi - -cat >>confdefs.h <<_ACEOF -#define HAVE_DECL_TZNAME $ac_have_decl -_ACEOF - - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5 -$as_echo_n "checking for tzname... " >&6; } -if ${ac_cv_var_tzname+:} false; then : - $as_echo_n "(cached) " >&6 -else - cat confdefs.h - <<_ACEOF >conftest.$ac_ext -/* end confdefs.h. */ -#include <time.h> -#if !HAVE_DECL_TZNAME -extern char *tzname[]; -#endif - -int -main () -{ -return tzname[0][0]; - ; - return 0; -} -_ACEOF -if ac_fn_c_try_link "$LINENO"; then : - ac_cv_var_tzname=yes -else - ac_cv_var_tzname=no -fi -rm -f core conftest.err conftest.$ac_objext \ - conftest$ac_exeext conftest.$ac_ext -fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_var_tzname" >&5 -$as_echo "$ac_cv_var_tzname" >&6; } - if test $ac_cv_var_tzname = yes; then - -$as_echo "#define HAVE_TZNAME 1" >>confdefs.h - - fi -fi - - # check tzset(3) exists and works like we expect it to { $as_echo "$as_me:${as_lineno-$LINENO}: checking for working tzset()" >&5 $as_echo_n "checking for working tzset()... " >&6; } diff --git a/configure.ac b/configure.ac index 913d546..aecf606 100644 --- a/configure.ac +++ b/configure.ac @@ -117,7 +117,7 @@ AC_ARG_ENABLE(universalsdk, [ case $enableval in yes) - # Locate the best usable SDK, see Mac/README.txt for more + # Locate the best usable SDK, see Mac/README for more # information enableval="`/usr/bin/xcodebuild -version -sdk macosx Path 2>/dev/null`" if ! ( echo $enableval | grep -E '\.sdk' 1>/dev/null ) @@ -1005,9 +1005,9 @@ HAS_GIT=no-repository fi if test $HAS_GIT = found then - GITVERSION="git -C \$(srcdir) rev-parse --short HEAD" - GITTAG="git -C \$(srcdir) describe --all --always --dirty" - GITBRANCH="git -C \$(srcdir) name-rev --name-only HEAD" + GITVERSION="git --git-dir \$(srcdir)/.git rev-parse --short HEAD" + GITTAG="git --git-dir \$(srcdir)/.git describe --all --always --dirty" + GITBRANCH="git --git-dir \$(srcdir)/.git name-rev --name-only HEAD" else GITVERSION="" GITTAG="" @@ -1455,6 +1455,13 @@ if test "$Py_LTO" = 'true' ; then esac ;; esac + + if test "$ac_cv_prog_cc_g" = "yes" + then + # bpo-30345: Add -g to LDFLAGS when compiling with LTO + # to get debug symbols. + LTOFLAGS="$LTOFLAGS -g" + fi fi @@ -3115,7 +3122,7 @@ AC_CHECK_FUNCS(alarm setitimer getitimer bind_textdomain_codeset chown \ gai_strerror getgroups getlogin getloadavg getpeername getpgid getpid \ getentropy \ getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \ - initgroups kill killpg lchmod lchown lstat mkfifo mknod mktime mmap \ + initgroups kill killpg lchown lstat mkfifo mknod mktime mmap \ mremap nice pathconf pause plock poll pthread_init \ putenv readlink realpath \ select sem_open sem_timedwait sem_getvalue sem_unlink setegid seteuid \ @@ -3127,6 +3134,13 @@ AC_CHECK_FUNCS(alarm setitimer getitimer bind_textdomain_codeset chown \ sysconf tcgetpgrp tcsetpgrp tempnam timegm times tmpfile tmpnam tmpnam_r \ truncate uname unsetenv utimes waitpid wait3 wait4 wcscoll _getpty) +# Force lchmod off for Linux. Linux disallows changing the mode of symbolic +# links. Some libc implementations have a stub lchmod implementation that always +# returns an error. +if test "$MACHDEP" != linux; then + AC_CHECK_FUNC(lchmod) +fi + # For some functions, having a definition is not sufficient, since # we want to take their address. AC_MSG_CHECKING(for chroot) @@ -4399,11 +4413,6 @@ then [Define if poll() sets errno on invalid file descriptors.]) fi -# Before we can test tzset, we need to check if struct tm has a tm_zone -# (which is not required by ISO C or UNIX spec) and/or if we support -# tzname[] -AC_STRUCT_TIMEZONE - # check tzset(3) exists and works like we expect it to AC_MSG_CHECKING(for working tzset()) AC_CACHE_VAL(ac_cv_working_tzset, [ diff --git a/pyconfig.h.in b/pyconfig.h.in index 11c4a66..f828677 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -439,9 +439,6 @@ /* Define to 1 if you have the 'lchflags' function. */ #undef HAVE_LCHFLAGS -/* Define to 1 if you have the `lchmod' function. */ -#undef HAVE_LCHMOD - /* Define to 1 if you have the `lchown' function. */ #undef HAVE_LCHOWN |