From 02011f1f5d3fae6eac0209b5faccc06079dd1b41 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 20 Oct 2015 00:32:14 -0500 Subject: first cut at removing default cache busters --- docs/api/static.rst | 9 --- docs/narr/assets.rst | 80 ++++++++++------------ pyramid/config/views.py | 20 ++---- pyramid/static.py | 84 +----------------------- pyramid/tests/test_config/test_views.py | 10 --- pyramid/tests/test_static.py | 113 +------------------------------- 6 files changed, 43 insertions(+), 273 deletions(-) diff --git a/docs/api/static.rst b/docs/api/static.rst index b6b279139..e6d6e4618 100644 --- a/docs/api/static.rst +++ b/docs/api/static.rst @@ -9,17 +9,8 @@ :members: :inherited-members: - .. autoclass:: PathSegmentCacheBuster - :members: - .. autoclass:: QueryStringCacheBuster :members: - .. autoclass:: PathSegmentMd5CacheBuster - :members: - - .. autoclass:: QueryStringMd5CacheBuster - :members: - .. autoclass:: QueryStringConstantCacheBuster :members: diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 020794062..397e0258d 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -372,30 +372,38 @@ assets by passing the optional argument, ``cachebust`` to .. code-block:: python :linenos: + import time + from pyramid.static import QueryStringConstantCacheBuster + # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=True) + config.add_static_view( + name='static', path='mypackage:folder/static', + cachebust=QueryStringConstantCacheBuster(str(int(time.time()))), + ) Setting the ``cachebust`` argument instructs :app:`Pyramid` to use a cache -busting scheme which adds the md5 checksum for a static asset as a path segment -in the asset's URL: +busting scheme which adds the curent time for a static asset to the query +string in the asset's URL: .. code-block:: python :linenos: js_url = request.static_url('mypackage:folder/static/js/myapp.js') - # Returns: 'http://www.example.com/static/c9658b3c0a314a1ca21e5988e662a09e/js/myapp.js' + # Returns: 'http://www.example.com/static/js/myapp.js?x=1445318121' -When the asset changes, so will its md5 checksum, and therefore so will its -URL. Supplying the ``cachebust`` argument also causes the static view to set -headers instructing clients to cache the asset for ten years, unless the -``cache_max_age`` argument is also passed, in which case that value is used. +When the web server restarts, the time constant will change and therefore so +will its URL. Supplying the ``cachebust`` argument also causes the static +view to set headers instructing clients to cache the asset for ten years, +unless the ``cache_max_age`` argument is also passed, in which case that +value is used. -.. note:: +.. warning:: - md5 checksums are cached in RAM, so if you change a static resource without - restarting your application, you may still generate URLs with a stale md5 - checksum. + Cache busting is an inherently complex topic as it integrates the asset + pipeline and the web application. It is expected and desired that + application authors will write their own cache buster implementations + conforming to the properties of their own asset pipelines. See + :ref:`custom_cache_busters` for information on writing your own. Disabling the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -406,40 +414,21 @@ configured cache busters without changing calls to ``PYRAMID_PREVENT_CACHEBUST`` environment variable or the ``pyramid.prevent_cachebust`` configuration value to a true value. +.. _custom_cache_busters: + Customizing the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Revisiting from the previous section: - -.. code-block:: python - :linenos: +The ``cachebust`` option to +:meth:`~pyramid.config.Configurator.add_static_view` may be set to any object +that implements the interface :class:`~pyramid.interfaces.ICacheBuster`. - # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=True) - -Setting ``cachebust`` to ``True`` instructs :app:`Pyramid` to use a default -cache busting implementation that should work for many situations. The -``cachebust`` may be set to any object that implements the interface -:class:`~pyramid.interfaces.ICacheBuster`. The above configuration is exactly -equivalent to: - -.. code-block:: python - :linenos: - - from pyramid.static import PathSegmentMd5CacheBuster - - # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=PathSegmentMd5CacheBuster()) - -:app:`Pyramid` includes a handful of ready to use cache buster implementations: -:class:`~pyramid.static.PathSegmentMd5CacheBuster`, which inserts an md5 -checksum token in the path portion of the asset's URL, -:class:`~pyramid.static.QueryStringMd5CacheBuster`, which adds an md5 checksum -token to the query string of the asset's URL, and +:app:`Pyramid` ships with a very simplistic :class:`~pyramid.static.QueryStringConstantCacheBuster`, which adds an -arbitrary token you provide to the query string of the asset's URL. +arbitrary token you provide to the query string of the asset's URL. This +is almost never what you want in production as it does not allow fine-grained +busting of individual assets. + In order to implement your own cache buster, you can write your own class from scratch which implements the :class:`~pyramid.interfaces.ICacheBuster` @@ -456,15 +445,16 @@ the hash of the currently checked out code: import os import subprocess - from pyramid.static import PathSegmentCacheBuster + from pyramid.static import QueryStringCacheBuster - class GitCacheBuster(PathSegmentCacheBuster): + class GitCacheBuster(QueryStringCacheBuster): """ Assuming your code is installed as a Git checkout, as opposed to an egg from an egg repository like PYPI, you can use this cachebuster to get the current commit's SHA1 to use as the cache bust token. """ - def __init__(self): + def __init__(self, param='x'): + super(GitCacheBuster, self).__init__(param=param) here = os.path.dirname(os.path.abspath(__file__)) self.sha1 = subprocess.check_output( ['git', 'rev-parse', 'HEAD'], diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 34fc49c00..e386bc4e1 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -34,7 +34,6 @@ from pyramid.interfaces import ( ) from pyramid import renderers -from pyramid.static import PathSegmentMd5CacheBuster from pyramid.compat import ( string_types, @@ -1862,14 +1861,12 @@ class ViewsConfiguratorMixin(object): The ``cachebust`` keyword argument may be set to cause :meth:`~pyramid.request.Request.static_url` to use cache busting when generating URLs. See :ref:`cache_busting` for general information - about cache busting. The value of the ``cachebust`` argument may be - ``True``, in which case a default cache busting implementation is used. - The value of the ``cachebust`` argument may also be an object which - implements :class:`~pyramid.interfaces.ICacheBuster`. See the - :mod:`~pyramid.static` module for some implementations. If the - ``cachebust`` argument is provided, the default for ``cache_max_age`` - is modified to be ten years. ``cache_max_age`` may still be explicitly - provided to override this default. + about cache busting. The value of the ``cachebust`` argument must + be an object which implements + :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` + argument is provided, the default for ``cache_max_age`` is modified + to be ten years. ``cache_max_age`` may still be explicitly provided + to override this default. The ``permission`` keyword argument is used to specify the :term:`permission` required by a user to execute the static view. By @@ -1967,9 +1964,6 @@ def isexception(o): @implementer(IStaticURLInfo) class StaticURLInfo(object): - # Indirection for testing - _default_cachebust = PathSegmentMd5CacheBuster - def _get_registrations(self, registry): try: reg = registry._static_url_registrations @@ -2033,8 +2027,6 @@ class StaticURLInfo(object): cb = None else: cb = extra.pop('cachebust', None) - if cb is True: - cb = self._default_cachebust() if cb: def cachebust(subpath, kw): subpath_tuple = tuple(subpath.split('/')) diff --git a/pyramid/static.py b/pyramid/static.py index cb78feb9b..2aff02c0c 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -1,5 +1,4 @@ # -*- coding: utf-8 -*- -import hashlib import os from os.path import ( @@ -27,7 +26,7 @@ from pyramid.httpexceptions import ( HTTPMovedPermanently, ) -from pyramid.path import AssetResolver, caller_package +from pyramid.path import caller_package from pyramid.response import FileResponse from pyramid.traversal import traversal_path_info @@ -159,71 +158,6 @@ def _secure_path(path_tuple): encoded = slash.join(path_tuple) # will be unicode return encoded -def _generate_md5(spec): - asset = AssetResolver(None).resolve(spec) - md5 = hashlib.md5() - with asset.stream() as stream: - for block in iter(lambda: stream.read(4096), b''): - md5.update(block) - return md5.hexdigest() - -class Md5AssetTokenGenerator(object): - """ - A mixin class which provides an implementation of - :meth:`~pyramid.interfaces.ICacheBuster.target` which generates an md5 - checksum token for an asset, caching it for subsequent calls. - """ - def __init__(self): - self.token_cache = {} - - def tokenize(self, pathspec): - # An astute observer will notice that this use of token_cache doesn't - # look particularly thread safe. Basic read/write operations on Python - # dicts, however, are atomic, so simply accessing and writing values - # to the dict shouldn't cause a segfault or other catastrophic failure. - # (See: http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm) - # - # We do have a race condition that could result in the same md5 - # checksum getting computed twice or more times in parallel. Since - # the program would still function just fine if this were to occur, - # the extra overhead of using locks to serialize access to the dict - # seems an unnecessary burden. - # - token = self.token_cache.get(pathspec) - if not token: - self.token_cache[pathspec] = token = _generate_md5(pathspec) - return token - -class PathSegmentCacheBuster(object): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which - inserts a token for cache busting in the path portion of an asset URL. - - To use this class, subclass it and provide a ``tokenize`` method which - accepts a ``pathspec`` and returns a token. - - .. versionadded:: 1.6 - """ - def pregenerate(self, pathspec, subpath, kw): - token = self.tokenize(pathspec) - return (token,) + subpath, kw - - def match(self, subpath): - return subpath[1:] - -class PathSegmentMd5CacheBuster(PathSegmentCacheBuster, - Md5AssetTokenGenerator): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which - inserts an md5 checksum token for cache busting in the path portion of an - asset URL. Generated md5 checksums are cached in order to speed up - subsequent calls. - - .. versionadded:: 1.6 - """ - def __init__(self): - super(PathSegmentMd5CacheBuster, self).__init__() - class QueryStringCacheBuster(object): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds @@ -249,22 +183,6 @@ class QueryStringCacheBuster(object): kw['_query'] = tuple(query) + ((self.param, token),) return subpath, kw -class QueryStringMd5CacheBuster(QueryStringCacheBuster, - Md5AssetTokenGenerator): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds - an md5 checksum token for cache busting in the query string of an asset - URL. Generated md5 checksums are cached in order to speed up subsequent - calls. - - The optional ``param`` argument determines the name of the parameter added - to the query string and defaults to ``'x'``. - - .. versionadded:: 1.6 - """ - def __init__(self, param='x'): - super(QueryStringMd5CacheBuster, self).__init__(param=param) - class QueryStringConstantCacheBuster(QueryStringCacheBuster): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index 1c2d300a1..acfb81962 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -4104,16 +4104,6 @@ class TestStaticURLInfo(unittest.TestCase): self.assertEqual(config.view_kw['renderer'], 'mypackage:templates/index.pt') - def test_add_cachebust_default(self): - config = self._makeConfig() - inst = self._makeOne() - inst._default_cachebust = lambda: DummyCacheBuster('foo') - inst.add(config, 'view', 'mypackage:path', cachebust=True) - cachebust = config.registry._static_url_registrations[0][3] - subpath, kw = cachebust('some/path', {}) - self.assertEqual(subpath, 'some/path') - self.assertEqual(kw['x'], 'foo') - def test_add_cachebust_prevented(self): config = self._makeConfig() config.registry.settings['pyramid.prevent_cachebust'] = True diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index a3df74b44..7f50a0e43 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -368,118 +368,7 @@ class Test_static_view_use_subpath_True(unittest.TestCase): from pyramid.httpexceptions import HTTPNotFound self.assertRaises(HTTPNotFound, inst, context, request) -class TestMd5AssetTokenGenerator(unittest.TestCase): - _fspath = None - _tmp = None - - @property - def fspath(self): - if self._fspath: - return self._fspath - - import os - import tempfile - self._tmp = tmp = tempfile.mkdtemp() - self._fspath = os.path.join(tmp, 'test.txt') - return self._fspath - - def tearDown(self): - import shutil - if self._tmp: - shutil.rmtree(self._tmp) - - def _makeOne(self): - from pyramid.static import Md5AssetTokenGenerator as cls - return cls() - - def test_package_resource(self): - fut = self._makeOne().tokenize - expected = '76d653a3a044e2f4b38bb001d283e3d9' - token = fut('pyramid.tests:fixtures/static/index.html') - self.assertEqual(token, expected) - - def test_filesystem_resource(self): - fut = self._makeOne().tokenize - expected = 'd5155f250bef0e9923e894dbc713c5dd' - with open(self.fspath, 'w') as f: - f.write("Are we rich yet?") - token = fut(self.fspath) - self.assertEqual(token, expected) - - def test_cache(self): - fut = self._makeOne().tokenize - expected = 'd5155f250bef0e9923e894dbc713c5dd' - with open(self.fspath, 'w') as f: - f.write("Are we rich yet?") - token = fut(self.fspath) - self.assertEqual(token, expected) - - # md5 shouldn't change because we've cached it - with open(self.fspath, 'w') as f: - f.write("Sorry for the convenience.") - token = fut(self.fspath) - self.assertEqual(token, expected) - -class TestPathSegmentMd5CacheBuster(unittest.TestCase): - - def _makeOne(self): - from pyramid.static import PathSegmentMd5CacheBuster as cls - inst = cls() - inst.tokenize = lambda pathspec: 'foo' - return inst - - def test_token(self): - fut = self._makeOne().tokenize - self.assertEqual(fut('whatever'), 'foo') - - def test_pregenerate(self): - fut = self._makeOne().pregenerate - self.assertEqual(fut('foo', ('bar',), 'kw'), (('foo', 'bar'), 'kw')) - - def test_match(self): - fut = self._makeOne().match - self.assertEqual(fut(('foo', 'bar')), ('bar',)) - -class TestQueryStringMd5CacheBuster(unittest.TestCase): - - def _makeOne(self, param=None): - from pyramid.static import QueryStringMd5CacheBuster as cls - if param: - inst = cls(param) - else: - inst = cls() - inst.tokenize = lambda pathspec: 'foo' - return inst - - def test_token(self): - fut = self._makeOne().tokenize - self.assertEqual(fut('whatever'), 'foo') - - def test_pregenerate(self): - fut = self._makeOne().pregenerate - self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'x': 'foo'}})) - - def test_pregenerate_change_param(self): - fut = self._makeOne('y').pregenerate - self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'y': 'foo'}})) - - def test_pregenerate_query_is_already_tuples(self): - fut = self._makeOne().pregenerate - self.assertEqual( - fut('foo', ('bar',), {'_query': [('a', 'b')]}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) - - def test_pregenerate_query_is_tuple_of_tuples(self): - fut = self._makeOne().pregenerate - self.assertEqual( - fut('foo', ('bar',), {'_query': (('a', 'b'),)}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) - -class TestQueryStringConstantCacheBuster(TestQueryStringMd5CacheBuster): +class TestQueryStringConstantCacheBuster(unittest.TestCase): def _makeOne(self, param=None): from pyramid.static import QueryStringConstantCacheBuster as cls -- cgit v1.2.3 From 315b19e8ea997f6b10c23dffa65d831aa42f3563 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Domen=20Ko=C5=BEar?= Date: Tue, 27 Oct 2015 10:58:47 +0100 Subject: pserve: track files with SyntaxError and skip reload --- pyramid/scripts/pserve.py | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 094a811a4..1184d7faa 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -14,6 +14,7 @@ import errno import logging import optparse import os +import py_compile import re import subprocess import sys @@ -820,7 +821,7 @@ class Monitor(object): # pragma: no cover if %errorlevel% == 3 goto repeat or run a monitoring process in Python (``pserve --reload`` does - this). + this). Use the ``watch_file(filename)`` function to cause a reload/restart for other non-Python files (e.g., configuration files). If you have @@ -843,6 +844,7 @@ class Monitor(object): # pragma: no cover self.poll_interval = poll_interval self.extra_files = list(self.global_extra_files) self.instances.append(self) + self.syntax_error_files = [] self.file_callbacks = list(self.global_file_callbacks) def _exit(self): @@ -887,11 +889,28 @@ class Monitor(object): # pragma: no cover continue if filename.endswith('.pyc') and os.path.exists(filename[:-1]): mtime = max(os.stat(filename[:-1]).st_mtime, mtime) + pyc = True + else: + pyc = False if filename not in self.module_mtimes: self.module_mtimes[filename] = mtime elif self.module_mtimes[filename] < mtime: - print("%s changed; reloading..." % filename) - return False + if pyc: + filename = filename[:-1] + # check if a file has syntax errors. If so, track it until it's fixed + for filename_promise in self.syntax_error_files + [filename]: + try: + py_compile.compile(filename_promise, doraise=True) + except py_compile.PyCompileError: + print("%s has a SyntaxError; NOT reloading." % filename_promise) + if filename_promise not in self.syntax_error_files: + self.syntax_error_files.append(filename_promise) + else: + if filename_promise in self.syntax_error_files: + self.syntax_error_files.remove(filename_promise) + if not self.syntax_error_files: + print("%s changed; reloading..." % filename) + return False return True def watch_file(self, cls, filename): -- cgit v1.2.3 From 565058df3e98ba45945b6254b5c69c3de399bcb7 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 28 Oct 2015 12:15:07 -0500 Subject: avoid spamming by outputting state only once when a file changes --- pyramid/scripts/pserve.py | 46 ++++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 1184d7faa..4a6e917a7 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -844,7 +844,8 @@ class Monitor(object): # pragma: no cover self.poll_interval = poll_interval self.extra_files = list(self.global_extra_files) self.instances.append(self) - self.syntax_error_files = [] + self.syntax_error_files = set() + self.pending_reload = False self.file_callbacks = list(self.global_file_callbacks) def _exit(self): @@ -892,25 +893,34 @@ class Monitor(object): # pragma: no cover pyc = True else: pyc = False - if filename not in self.module_mtimes: - self.module_mtimes[filename] = mtime - elif self.module_mtimes[filename] < mtime: + old_mtime = self.module_mtimes.get(filename) + self.module_mtimes[filename] = mtime + if old_mtime is not None and old_mtime < mtime: + self.pending_reload = True if pyc: filename = filename[:-1] - # check if a file has syntax errors. If so, track it until it's fixed - for filename_promise in self.syntax_error_files + [filename]: - try: - py_compile.compile(filename_promise, doraise=True) - except py_compile.PyCompileError: - print("%s has a SyntaxError; NOT reloading." % filename_promise) - if filename_promise not in self.syntax_error_files: - self.syntax_error_files.append(filename_promise) - else: - if filename_promise in self.syntax_error_files: - self.syntax_error_files.remove(filename_promise) - if not self.syntax_error_files: - print("%s changed; reloading..." % filename) - return False + is_valid = True + if filename.endswith('.py'): + is_valid = self.check_syntax(filename) + if is_valid: + print("%s changed ..." % filename) + if self.pending_reload and not self.syntax_error_files: + self.pending_reload = False + return False + return True + + def check_syntax(self, filename): + # check if a file has syntax errors. + # If so, track it until it's fixed. + try: + py_compile.compile(filename, doraise=True) + except py_compile.PyCompileError: + print("%s has a SyntaxError; NOT reloading." % filename) + self.syntax_error_files.add(filename) + return False + else: + if filename in self.syntax_error_files: + self.syntax_error_files.remove(filename) return True def watch_file(self, cls, filename): -- cgit v1.2.3 From 67ff3b9ce4bcaceeee90f2aa4ae900d9220cac8b Mon Sep 17 00:00:00 2001 From: RamiC Date: Wed, 28 Oct 2015 21:38:22 +0200 Subject: Convert timeout argument to int values when applicable --- CONTRIBUTORS.txt | 2 ++ pyramid/session.py | 2 +- pyramid/tests/test_session.py | 16 ++++++++++++++++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt index 2ef07af75..4edf1b4e9 100644 --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -254,3 +254,5 @@ Contributors - Jesse Dhillon, 2015/10/07 - Amos Latteier, 2015/10/22 + +- Rami Chousein, 2015/10/28 diff --git a/pyramid/session.py b/pyramid/session.py index c4cfc1949..525636688 100644 --- a/pyramid/session.py +++ b/pyramid/session.py @@ -244,7 +244,7 @@ def BaseCookieSessionFactory( _cookie_secure = secure _cookie_httponly = httponly _cookie_on_exception = set_on_exception - _timeout = timeout + _timeout = timeout if timeout is None else int(timeout) _reissue_time = reissue_time # dirty flag diff --git a/pyramid/tests/test_session.py b/pyramid/tests/test_session.py index eac6593d9..c3a1f3055 100644 --- a/pyramid/tests/test_session.py +++ b/pyramid/tests/test_session.py @@ -62,6 +62,22 @@ class SharedCookieSessionTests(object): session = self._makeOne(request, timeout=None) self.assertEqual(dict(session), {'state': 1}) + def test_timeout_str(self): + import time + request = testing.DummyRequest() + cookieval = self._serialize((time.time() - 5, 0, {'state': 1})) + request.cookies['session'] = cookieval + session = self._makeOne(request, timeout='1') + self.assertEqual(dict(session), {}) + + def test_timeout_invalid(self): + import time + request = testing.DummyRequest() + cookieval = self._serialize((time.time() - 5, 0, {'state': 1})) + request.cookies['session'] = cookieval + self.assertRaises(ValueError, self._makeOne, request, timeout='error') + + def test_changed(self): request = testing.DummyRequest() session = self._makeOne(request) -- cgit v1.2.3 From a43abd25858d3e60f525c12da5a271e30a99c4fb Mon Sep 17 00:00:00 2001 From: RamiC Date: Thu, 29 Oct 2015 19:32:58 +0200 Subject: Convert reissue_time argument to int values when applicable --- pyramid/session.py | 3 ++- pyramid/tests/test_session.py | 27 ++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/pyramid/session.py b/pyramid/session.py index 525636688..56140b4c7 100644 --- a/pyramid/session.py +++ b/pyramid/session.py @@ -245,7 +245,7 @@ def BaseCookieSessionFactory( _cookie_httponly = httponly _cookie_on_exception = set_on_exception _timeout = timeout if timeout is None else int(timeout) - _reissue_time = reissue_time + _reissue_time = reissue_time if reissue_time is None else int(reissue_time) # dirty flag _dirty = False @@ -390,6 +390,7 @@ def BaseCookieSessionFactory( def UnencryptedCookieSessionFactoryConfig( secret, timeout=1200, + reissue_time=0, cookie_name='session', cookie_max_age=None, cookie_path='/', diff --git a/pyramid/tests/test_session.py b/pyramid/tests/test_session.py index c3a1f3055..147ce7fa1 100644 --- a/pyramid/tests/test_session.py +++ b/pyramid/tests/test_session.py @@ -77,7 +77,6 @@ class SharedCookieSessionTests(object): request.cookies['session'] = cookieval self.assertRaises(ValueError, self._makeOne, request, timeout='error') - def test_changed(self): request = testing.DummyRequest() session = self._makeOne(request) @@ -313,6 +312,19 @@ class TestBaseCookieSession(SharedCookieSessionTests, unittest.TestCase): self.assertEqual(session['state'], 1) self.assertFalse(session._dirty) + def test_reissue_str_triggered(self): + import time + request = testing.DummyRequest() + cookieval = self._serialize((time.time() - 2, 0, {'state': 1})) + request.cookies['session'] = cookieval + session = self._makeOne(request, reissue_time='0') + self.assertEqual(session['state'], 1) + self.assertTrue(session._dirty) + + def test_reissue_invalid(self): + request = testing.DummyRequest() + self.assertRaises(ValueError, self._makeOne, request, reissue_time='invalid value') + class TestSignedCookieSession(SharedCookieSessionTests, unittest.TestCase): def _makeOne(self, request, **kw): from pyramid.session import SignedCookieSessionFactory @@ -347,6 +359,19 @@ class TestSignedCookieSession(SharedCookieSessionTests, unittest.TestCase): self.assertEqual(session['state'], 1) self.assertFalse(session._dirty) + def test_reissue_str_triggered(self): + import time + request = testing.DummyRequest() + cookieval = self._serialize((time.time() - 2, 0, {'state': 1})) + request.cookies['session'] = cookieval + session = self._makeOne(request, reissue_time='0') + self.assertEqual(session['state'], 1) + self.assertTrue(session._dirty) + + def test_reissue_invalid(self): + request = testing.DummyRequest() + self.assertRaises(ValueError, self._makeOne, request, reissue_time='invalid value') + def test_custom_salt(self): import time request = testing.DummyRequest() -- cgit v1.2.3 From 7ca2b205cd2079f4e5c9b03aec82ac75cfa8fe88 Mon Sep 17 00:00:00 2001 From: RamiC Date: Thu, 29 Oct 2015 19:38:45 +0200 Subject: Remove unnecessary code --- pyramid/tests/test_session.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/pyramid/tests/test_session.py b/pyramid/tests/test_session.py index 147ce7fa1..68741b265 100644 --- a/pyramid/tests/test_session.py +++ b/pyramid/tests/test_session.py @@ -71,11 +71,8 @@ class SharedCookieSessionTests(object): self.assertEqual(dict(session), {}) def test_timeout_invalid(self): - import time request = testing.DummyRequest() - cookieval = self._serialize((time.time() - 5, 0, {'state': 1})) - request.cookies['session'] = cookieval - self.assertRaises(ValueError, self._makeOne, request, timeout='error') + self.assertRaises(ValueError, self._makeOne, request, timeout='Invalid value') def test_changed(self): request = testing.DummyRequest() -- cgit v1.2.3 From b6f0be8bba239fbc1a3104c530ec996d8a5ee62e Mon Sep 17 00:00:00 2001 From: RamiC Date: Thu, 29 Oct 2015 19:46:51 +0200 Subject: Remove a leaked line from a local test --- pyramid/session.py | 1 - 1 file changed, 1 deletion(-) diff --git a/pyramid/session.py b/pyramid/session.py index 56140b4c7..18aa41a0f 100644 --- a/pyramid/session.py +++ b/pyramid/session.py @@ -390,7 +390,6 @@ def BaseCookieSessionFactory( def UnencryptedCookieSessionFactoryConfig( secret, timeout=1200, - reissue_time=0, cookie_name='session', cookie_max_age=None, cookie_path='/', -- cgit v1.2.3 From fa7886f89b99b52bcbed6b7d3577e7f1caace3df Mon Sep 17 00:00:00 2001 From: RamiC Date: Sat, 31 Oct 2015 09:56:28 +0200 Subject: Convert max_age argument to int when applicable --- pyramid/session.py | 2 +- pyramid/tests/test_session.py | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/pyramid/session.py b/pyramid/session.py index 18aa41a0f..fa85fe69c 100644 --- a/pyramid/session.py +++ b/pyramid/session.py @@ -238,7 +238,7 @@ def BaseCookieSessionFactory( # configuration parameters _cookie_name = cookie_name - _cookie_max_age = max_age + _cookie_max_age = max_age if max_age is None else int(max_age) _cookie_path = path _cookie_domain = domain _cookie_secure = secure diff --git a/pyramid/tests/test_session.py b/pyramid/tests/test_session.py index 68741b265..82e4fb001 100644 --- a/pyramid/tests/test_session.py +++ b/pyramid/tests/test_session.py @@ -322,6 +322,10 @@ class TestBaseCookieSession(SharedCookieSessionTests, unittest.TestCase): request = testing.DummyRequest() self.assertRaises(ValueError, self._makeOne, request, reissue_time='invalid value') + def test_cookie_max_age_invalid(self): + request = testing.DummyRequest() + self.assertRaises(ValueError, self._makeOne, request, max_age='invalid value') + class TestSignedCookieSession(SharedCookieSessionTests, unittest.TestCase): def _makeOne(self, request, **kw): from pyramid.session import SignedCookieSessionFactory @@ -369,6 +373,10 @@ class TestSignedCookieSession(SharedCookieSessionTests, unittest.TestCase): request = testing.DummyRequest() self.assertRaises(ValueError, self._makeOne, request, reissue_time='invalid value') + def test_cookie_max_age_invalid(self): + request = testing.DummyRequest() + self.assertRaises(ValueError, self._makeOne, request, max_age='invalid value') + def test_custom_salt(self): import time request = testing.DummyRequest() -- cgit v1.2.3 From e2519d64b3dd1f53555a184eb38b5138ea9bc2f6 Mon Sep 17 00:00:00 2001 From: RamiC Date: Tue, 3 Nov 2015 10:20:43 +0200 Subject: Convert AuthTktCookieHelper time related parameters to int when applicable --- pyramid/authentication.py | 9 ++++--- pyramid/tests/test_authentication.py | 46 ++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/pyramid/authentication.py b/pyramid/authentication.py index 0924b5901..9bf1de62e 100644 --- a/pyramid/authentication.py +++ b/pyramid/authentication.py @@ -855,9 +855,9 @@ class AuthTktCookieHelper(object): self.cookie_name = cookie_name self.secure = secure self.include_ip = include_ip - self.timeout = timeout - self.reissue_time = reissue_time - self.max_age = max_age + self.timeout = timeout if timeout is None else int(timeout) + self.reissue_time = reissue_time if reissue_time is None else int(reissue_time) + self.max_age = max_age if max_age is None else int(max_age) self.wild_domain = wild_domain self.parent_domain = parent_domain self.domain = domain @@ -977,8 +977,7 @@ class AuthTktCookieHelper(object): Tokens are available in the returned identity when an auth_tkt is found in the request and unpacked. Default: ``()``. """ - if max_age is None: - max_age = self.max_age + max_age = self.max_age if max_age is None else int(max_age) environ = request.environ diff --git a/pyramid/tests/test_authentication.py b/pyramid/tests/test_authentication.py index c7fc1c211..595a0eac8 100644 --- a/pyramid/tests/test_authentication.py +++ b/pyramid/tests/test_authentication.py @@ -600,6 +600,15 @@ class TestAuthTktCookieHelper(unittest.TestCase): cookies.load(cookie) return cookies.get('auth_tkt') + def test_init_cookie_str_reissue_invalid(self): + self.assertRaises(ValueError, self._makeOne, 'secret', reissue_time='invalid value') + + def test_init_cookie_str_timeout_invalid(self): + self.assertRaises(ValueError, self._makeOne, 'secret', timeout='invalid value') + + def test_init_cookie_str_max_age_invalid(self): + self.assertRaises(ValueError, self._makeOne, 'secret', max_age='invalid value') + def test_identify_nocookie(self): helper = self._makeOne('secret') request = self._makeRequest() @@ -758,6 +767,12 @@ class TestAuthTktCookieHelper(unittest.TestCase): result = helper.identify(request) self.assertEqual(result, None) + def test_identify_cookie_str_timeout(self): + helper = self._makeOne('secret', timeout='1') + request = self._makeRequest({'HTTP_COOKIE':'auth_tkt=bogus'}) + result = helper.identify(request) + self.assertEqual(result, None) + def test_identify_cookie_reissue(self): import time helper = self._makeOne('secret', timeout=10, reissue_time=0) @@ -774,6 +789,22 @@ class TestAuthTktCookieHelper(unittest.TestCase): self.assertEqual(len(response.headerlist), 3) self.assertEqual(response.headerlist[0][0], 'Set-Cookie') + def test_identify_cookie_str_reissue(self): + import time + helper = self._makeOne('secret', timeout=10, reissue_time='0') + now = time.time() + helper.auth_tkt.timestamp = now + helper.now = now + 1 + helper.auth_tkt.tokens = (text_('a'), ) + request = self._makeRequest('bogus') + result = helper.identify(request) + self.assertTrue(result) + self.assertEqual(len(request.callbacks), 1) + response = DummyResponse() + request.callbacks[0](request, response) + self.assertEqual(len(response.headerlist), 3) + self.assertEqual(response.headerlist[0][0], 'Set-Cookie') + def test_identify_cookie_reissue_already_reissued_this_request(self): import time helper = self._makeOne('secret', timeout=10, reissue_time=0) @@ -1058,6 +1089,16 @@ class TestAuthTktCookieHelper(unittest.TestCase): self.assertTrue('userid' in value.value) def test_remember_max_age(self): + helper = self._makeOne('secret') + request = self._makeRequest() + result = helper.remember(request, 'userid', max_age=500) + values = self._parseHeaders(result) + self.assertEqual(len(result), 3) + + self.assertEqual(values[0]['max-age'], '500') + self.assertTrue(values[0]['expires']) + + def test_remember_str_max_age(self): helper = self._makeOne('secret') request = self._makeRequest() result = helper.remember(request, 'userid', max_age='500') @@ -1067,6 +1108,11 @@ class TestAuthTktCookieHelper(unittest.TestCase): self.assertEqual(values[0]['max-age'], '500') self.assertTrue(values[0]['expires']) + def test_remember_str_max_age_invalid(self): + helper = self._makeOne('secret') + request = self._makeRequest() + self.assertRaises(ValueError, helper.remember, request, 'userid', max_age='invalid value') + def test_remember_tokens(self): helper = self._makeOne('secret') request = self._makeRequest() -- cgit v1.2.3 From 81cd59826279dab5959a3e198eee75ab09910872 Mon Sep 17 00:00:00 2001 From: RamiC Date: Tue, 3 Nov 2015 10:22:07 +0200 Subject: Include CHANGES entry for converting time related parameters of AuthTktCookieHelper and CookieSession --- CHANGES.txt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 8b63cf847..aefb166dc 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -232,6 +232,11 @@ Bug Fixes shell a little more straightfoward. See https://github.com/Pylons/pyramid/pull/1883 +- Fix an issue when user passes unparsed strings to ``pyramid.session.CookieSession`` + and ``pyramid.authentication.AuthTktCookieHelper`` for time related parameters + ``timeout``, ``reissue_time``, ``max_age`` that expect an integer value. + See https://github.com/Pylons/pyramid/pull/2050 + Deprecations ------------ -- cgit v1.2.3 From 891d05499e54c067863a4fae9f2301ab772761dc Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 00:20:27 -0600 Subject: add manifest cache buster --- pyramid/static.py | 82 +++++++++++++++++++++++++++++++++++ pyramid/tests/fixtures/manifest.json | 4 ++ pyramid/tests/fixtures/manifest2.json | 4 ++ pyramid/tests/test_static.py | 54 +++++++++++++++++++++++ 4 files changed, 144 insertions(+) create mode 100644 pyramid/tests/fixtures/manifest.json create mode 100644 pyramid/tests/fixtures/manifest2.json diff --git a/pyramid/static.py b/pyramid/static.py index 2aff02c0c..59f440c82 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -1,7 +1,9 @@ # -*- coding: utf-8 -*- +import json import os from os.path import ( + getmtime, normcase, normpath, join, @@ -202,3 +204,83 @@ class QueryStringConstantCacheBuster(QueryStringCacheBuster): def tokenize(self, pathspec): return self._token + +class ManifestCacheBuster(object): + """ + An implementation of :class:`~pyramid.interfaces.ICacheBuster` which + uses a supplied manifest file to map an asset path to a cache-busted + version of the path. + + The file is expected to conform to the following simple JSON format: + + .. code-block:: json + + { + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png", + } + + Specifically, it is a JSON-serialized dictionary where the keys are the + source asset paths used in calls to + :meth:`~pyramid.request.Request.static_url. For example:: + + .. code-block:: python + + >>> request.static_url('myapp:static/css/main.css') + "http://www.example.com/static/css/main-678b7c80.css" + + If a path is not found in the manifest it will pass through unchanged. + + If ``reload`` is ``True`` then the manifest file will be reloaded when + changed. It is not recommended to leave this enabled in production. + + If the manifest file cannot be found on disk it will be treated as + an empty mapping unless ``reload`` is ``False``. + + The default implementation assumes the requested (possibly cache-busted) + path is the actual filename on disk. Subclasses may override the ``match`` + method to alter this behavior. For example, to strip the cache busting + token from the path. + + .. versionadded:: 1.6 + """ + exists = staticmethod(exists) # testing + getmtime = staticmethod(getmtime) # testing + + def __init__(self, manifest_path, reload=False): + self.manifest_path = manifest_path + self.reload = reload + + self._mtime = None + if not reload: + self._manifest = self.parse_manifest() + + def parse_manifest(self): + """ + Return a mapping parsed from the ``manifest_path``. + + Subclasses may override this method to use something other than + ``json.loads``. + + """ + with open(self.manifest_path, 'rb') as fp: + content = fp.read().decode('utf-8') + return json.loads(content) + + @property + def manifest(self): + """ The current manifest dictionary.""" + if self.reload: + if not self.exists(self.manifest_path): + return {} + mtime = self.getmtime(self.manifest_path) + if self._mtime is None or mtime > self._mtime: + self._manifest = self.parse_manifest() + self._mtime = mtime + return self._manifest + + def pregenerate(self, pathspec, subpath, kw): + path = '/'.join(subpath) + path = self.manifest.get(path, path) + new_subpath = path.split('/') + return (new_subpath, kw) diff --git a/pyramid/tests/fixtures/manifest.json b/pyramid/tests/fixtures/manifest.json new file mode 100644 index 000000000..0a43bc5e3 --- /dev/null +++ b/pyramid/tests/fixtures/manifest.json @@ -0,0 +1,4 @@ +{ + "css/main.css": "css/main-test.css", + "images/background.png": "images/background-a8169106.png" +} diff --git a/pyramid/tests/fixtures/manifest2.json b/pyramid/tests/fixtures/manifest2.json new file mode 100644 index 000000000..fd6b9a7bb --- /dev/null +++ b/pyramid/tests/fixtures/manifest2.json @@ -0,0 +1,4 @@ +{ + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png" +} diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index 7f50a0e43..ac30e9e50 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -1,6 +1,9 @@ import datetime +import os.path import unittest +here = os.path.dirname(__file__) + # 5 years from now (more or less) fiveyrsfuture = datetime.datetime.utcnow() + datetime.timedelta(5*365) @@ -406,6 +409,57 @@ class TestQueryStringConstantCacheBuster(unittest.TestCase): fut('foo', ('bar',), {'_query': (('a', 'b'),)}), (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) +class TestManifestCacheBuster(unittest.TestCase): + + def _makeOne(self, path, **kw): + from pyramid.static import ManifestCacheBuster as cls + return cls(path, **kw) + + def test_it(self): + manifest_path = os.path.join(here, 'fixtures', 'manifest.json') + fut = self._makeOne(manifest_path).pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + + def test_reload(self): + manifest_path = os.path.join(here, 'fixtures', 'manifest.json') + new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') + inst = self._makeOne('foo', reload=True) + inst.getmtime = lambda *args, **kwargs: 0 + fut = inst.pregenerate + + # test without a valid manifest + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main.css'], {})) + + # swap to a real manifest, setting mtime to 0 + inst.manifest_path = manifest_path + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + + # ensure switching the path doesn't change the result + inst.manifest_path = new_manifest_path + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + + # update mtime, should cause a reload + inst.getmtime = lambda *args, **kwargs: 1 + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-678b7c80.css'], {})) + + def test_invalid_manifest(self): + self.assertRaises(IOError, lambda: self._makeOne('foo')) + + def test_invalid_manifest_with_reload(self): + inst = self._makeOne('foo', reload=True) + self.assertEqual(inst.manifest, {}) + class DummyContext: pass -- cgit v1.2.3 From 31f3d86d3bf5db3c4aa5085c7b7a4d6396f29931 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 00:55:38 -0600 Subject: complete cache buster docs using manifest example --- docs/api/static.rst | 3 + docs/glossary.rst | 4 ++ docs/narr/assets.rst | 183 ++++++++++++++++++++++----------------------------- 3 files changed, 84 insertions(+), 106 deletions(-) diff --git a/docs/api/static.rst b/docs/api/static.rst index e6d6e4618..f3727e197 100644 --- a/docs/api/static.rst +++ b/docs/api/static.rst @@ -9,6 +9,9 @@ :members: :inherited-members: + .. autoclass:: ManifestCacheBuster + :members: + .. autoclass:: QueryStringCacheBuster :members: diff --git a/docs/glossary.rst b/docs/glossary.rst index 9c0ea8598..b4bb36421 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -1089,3 +1089,7 @@ Glossary data in a Redis database. See https://pypi.python.org/pypi/pyramid_redis_sessions for more information. + cache busting + A technique used when serving a cacheable static asset in order to force + a client to query the new version of the asset. See :ref:`cache_busting` + for more information. diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 397e0258d..d36fa49c0 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -356,14 +356,14 @@ may change, and then you'll want the client to load a new copy of the asset. Under normal circumstances you'd just need to wait for the client's cached copy to expire before they get the new version of the static resource. -A commonly used workaround to this problem is a technique known as "cache -busting". Cache busting schemes generally involve generating a URL for a -static asset that changes when the static asset changes. This way headers can -be sent along with the static asset instructing the client to cache the asset -for a very long time. When a static asset is changed, the URL used to refer to -it in a web page also changes, so the client sees it as a new resource and -requests the asset, regardless of any caching policy set for the resource's old -URL. +A commonly used workaround to this problem is a technique known as +:term:`cache busting`. Cache busting schemes generally involve generating a +URL for a static asset that changes when the static asset changes. This way +headers can be sent along with the static asset instructing the client to cache +the asset for a very long time. When a static asset is changed, the URL used +to refer to it in a web page also changes, so the client sees it as a new +resource and requests the asset, regardless of any caching policy set for the +resource's old URL. :app:`Pyramid` can be configured to produce cache busting URLs for static assets by passing the optional argument, ``cachebust`` to @@ -397,7 +397,7 @@ view to set headers instructing clients to cache the asset for ten years, unless the ``cache_max_age`` argument is also passed, in which case that value is used. -.. warning:: +.. note:: Cache busting is an inherently complex topic as it integrates the asset pipeline and the web application. It is expected and desired that @@ -429,16 +429,14 @@ arbitrary token you provide to the query string of the asset's URL. This is almost never what you want in production as it does not allow fine-grained busting of individual assets. - In order to implement your own cache buster, you can write your own class from scratch which implements the :class:`~pyramid.interfaces.ICacheBuster` interface. Alternatively you may choose to subclass one of the existing implementations. One of the most likely scenarios is you'd want to change the -way the asset token is generated. To do this just subclass either -:class:`~pyramid.static.PathSegmentCacheBuster` or +way the asset token is generated. To do this just subclass :class:`~pyramid.static.QueryStringCacheBuster` and define a -``tokenize(pathspec)`` method. Here is an example which just uses Git to get -the hash of the currently checked out code: +``tokenize(pathspec)`` method. Here is an example which uses Git to get +the hash of the current commit: .. code-block:: python :linenos: @@ -466,26 +464,60 @@ the hash of the currently checked out code: Choosing a Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~ -The default cache buster implementation, -:class:`~pyramid.static.PathSegmentMd5CacheBuster`, works very well assuming -that you're using :app:`Pyramid` to serve your static assets. The md5 checksum -is fine grained enough that browsers should only request new versions of -specific assets that have changed. Many caching HTTP proxies will fail to -cache a resource if the URL contains a query string. In general, therefore, -you should prefer a cache busting strategy which modifies the path segment to a -strategy which adds a query string. - -It is possible, however, that your static assets are being served by another -web server or externally on a CDN. In these cases modifying the path segment -for a static asset URL would cause the external service to fail to find the -asset, causing your customer to get a 404. In these cases you would need to -fall back to a cache buster which adds a query string. It is even possible -that there isn't a copy of your static assets available to the :app:`Pyramid` -application, so a cache busting implementation that generates md5 checksums -would fail since it can't access the assets. In such a case, -:class:`~pyramid.static.QueryStringConstantCacheBuster` is a reasonable -fallback. The following code would set up a cachebuster that just uses the -time at start up as a cachebust token: +Many caching HTTP proxies will fail to cache a resource if the URL contains +a query string. Therefore, in general, you should prefer a cache busting +strategy which modifies the path segment rather than methods which add a +token to the query string. + +You will need to consider whether the :app:`Pyramid` application will be +serving your static assets, whether you are using an external asset pipeline +to handle rewriting urls internal to the css/javascript, and how fine-grained +do you want the cache busting tokens to be. + +In many cases you will want to host the static assets on another web server +or externally on a CDN. In these cases your :app:`Pyramid` application may not +even have access to a copy of the static assets. In order to cache bust these +assets you will need some information about them. + +If you are using an external asset pipeline to generate your static files you +should consider using the :class:`~pyramid.static.ManifestCacheBuster`. +This cache buster can load a standard JSON formatted file generated by your +pipeline and use it to cache bust the assets. This has many performance +advantages as :app:`Pyramid` does not need to look at the files to generate +any cache busting tokens, but still supports fine-grained per-file tokens. + +Assuming an example ``manifest.json`` like: + +.. code-block:: json + + { + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png" + } + +The following code would set up a cachebuster: + +.. code-block:: python + :linenos: + + from pyramid.path import AssetResolver + from pyramid.static import ManifestCacheBuster + + resolver = AssetResolver() + manifest = resolver.resolve('myapp:static/manifest.json') + config.add_static_view( + name='http://mycdn.example.com/', + path='mypackage:static', + cachebust=ManifestCacheBuster(manifest.abspath())) + +A simpler approach is to use the +:class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global +token that will bust all of the assets at once. The advantage of this strategy +is that it is simple and by using the query string there doesn't need to be +any shared information between your application and the static assets. + +The following code would set up a cachebuster that just uses the time at +start up as a cachebust token: .. code-block:: python :linenos: @@ -496,7 +528,7 @@ time at start up as a cachebust token: config.add_static_view( name='http://mycdn.example.com/', path='mypackage:static', - cachebust=QueryStringConstantCacheBuster(str(time.time()))) + cachebust=QueryStringConstantCacheBuster(str(int(time.time())))) .. index:: single: static assets view @@ -508,85 +540,24 @@ Often one needs to refer to images and other static assets inside CSS and JavaScript files. If cache busting is active, the final static asset URL is not available until the static assets have been assembled. These URLs cannot be handwritten. Thus, when having static asset references in CSS and JavaScript, -one needs to perform one of the following tasks. +one needs to perform one of the following tasks: * Process the files by using a precompiler which rewrites URLs to their final - cache busted form. + cache busted form. Then, you can use the + :class:`~pyramid.static.ManifestCacheBuster` to synchronize your asset + pipeline with :app:`Pyramid`, allowing the pipeline to have full control + over the final URLs of your assets. * Templatize JS and CSS, and call ``request.static_url()`` inside their template code. * Pass static URL references to CSS and JavaScript via other means. -Below are some simple approaches for CSS and JS programming which consider -asset cache busting. These approaches do not require additional tools or -packages. - -Relative cache busted URLs in CSS -+++++++++++++++++++++++++++++++++ - -Consider a CSS file ``/static/theme/css/site.css`` which contains the following -CSS code. - -.. code-block:: css - - body { - background: url(/static/theme/img/background.jpg); - } - -Any changes to ``background.jpg`` would not appear to the visitor because the -URL path is not cache busted as it is. Instead we would have to construct an -URL to the background image with the default ``PathSegmentCacheBuster`` cache -busting mechanism:: - - https://site/static/1eeb262c717/theme/img/background.jpg - -Every time the image is updated, the URL would need to be changed. It is not -practical to write this non-human readable URL into a CSS file. - -However, the CSS file itself is cache busted and is located under the path for -static assets. This lets us use relative references in our CSS to cache bust -the image. - -.. code-block:: css - - body { - background: url(../img/background.jpg); - } - -The browser would interpret this as having the CSS file hash in URL:: - - https://site/static/ab234b262c71/theme/css/../img/background.jpg - -The downside of this approach is that if the background image changes, one -needs to bump the CSS file. The CSS file hash change signals the caches that -the relative URL to the image in the CSS has been changed. When updating CSS -and related image assets, updates usually happen hand in hand, so this does not -add extra effort to theming workflow. - -Passing cache busted URLs to JavaScript -+++++++++++++++++++++++++++++++++++++++ - -For JavaScript, one can pass static asset URLs as function arguments or -globals. The globals can be generated in page template code, having access to -the ``request.static_url()`` function. - -Below is a simple example of passing a cached busted image URL in the Jinja2 -template language. Put the following code into the ```` section of the -relevant page. - -.. code-block:: html - - - -Then in your main ``site.js`` file, put the following code. - -.. code-block:: javascript - - var image = new Image(window.assets.backgroundImage); +If your CSS and JavaScript assets use URLs to reference other assets it is +recommended that you implement an external asset pipeline that can rewrite the +generated static files with new URLs containing cache busting tokens. The +machinery inside :app:`Pyramid` will not help with this step as it has very +little knowledge of the asset types your application may use. .. _advanced_static: -- cgit v1.2.3 From cbec33b898efffbfa6acaf91cae45ec0daed4d7a Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 19:53:28 -0600 Subject: update changelog with #2013 --- CHANGES.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGES.txt b/CHANGES.txt index 8b63cf847..18e8ba39c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -57,7 +57,8 @@ Features Core APIs are shipped for both cache busting via query strings and path segments and may be extended to fit into custom asset pipelines. See https://github.com/Pylons/pyramid/pull/1380 and - https://github.com/Pylons/pyramid/pull/1583 + https://github.com/Pylons/pyramid/pull/1583 and + https://github.com/Pylons/pyramid/pull/2013 - Add ``pyramid.config.Configurator.root_package`` attribute and init parameter to assist with includeable packages that wish to resolve -- cgit v1.2.3 From 7410250313f893e5952bb2697324a4d4e3d47d22 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 10:45:47 -0600 Subject: support asset specs in ManifestCacheBuster --- pyramid/static.py | 22 ++++++++++++++++------ pyramid/tests/test_static.py | 14 ++++++++++++++ 2 files changed, 30 insertions(+), 6 deletions(-) diff --git a/pyramid/static.py b/pyramid/static.py index 59f440c82..c2c8c89e5 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -19,7 +19,10 @@ from pkg_resources import ( from repoze.lru import lru_cache -from pyramid.asset import resolve_asset_spec +from pyramid.asset import ( + abspath_from_asset_spec, + resolve_asset_spec, +) from pyramid.compat import text_ @@ -211,7 +214,11 @@ class ManifestCacheBuster(object): uses a supplied manifest file to map an asset path to a cache-busted version of the path. - The file is expected to conform to the following simple JSON format: + The ``manifest_spec`` can be an absolute path or a :term:`asset spec` + pointing to a package-relative file. + + The manifest file is expected to conform to the following simple JSON + format: .. code-block:: json @@ -222,7 +229,7 @@ class ManifestCacheBuster(object): Specifically, it is a JSON-serialized dictionary where the keys are the source asset paths used in calls to - :meth:`~pyramid.request.Request.static_url. For example:: + :meth:`~pyramid.request.Request.static_url`. For example:: .. code-block:: python @@ -247,8 +254,10 @@ class ManifestCacheBuster(object): exists = staticmethod(exists) # testing getmtime = staticmethod(getmtime) # testing - def __init__(self, manifest_path, reload=False): - self.manifest_path = manifest_path + def __init__(self, manifest_spec, reload=False): + package_name = caller_package().__name__ + self.manifest_path = abspath_from_asset_spec( + manifest_spec, package_name) self.reload = reload self._mtime = None @@ -260,7 +269,8 @@ class ManifestCacheBuster(object): Return a mapping parsed from the ``manifest_path``. Subclasses may override this method to use something other than - ``json.loads``. + ``json.loads`` to load any type of file format and return a conforming + dictionary. """ with open(self.manifest_path, 'rb') as fp: diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index ac30e9e50..4a07c2cb1 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -423,6 +423,20 @@ class TestManifestCacheBuster(unittest.TestCase): fut('foo', ('css', 'main.css'), {}), (['css', 'main-test.css'], {})) + def test_it_with_relspec(self): + fut = self._makeOne('fixtures/manifest.json').pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + + def test_it_with_absspec(self): + fut = self._makeOne('pyramid.tests:fixtures/manifest.json').pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + def test_reload(self): manifest_path = os.path.join(here, 'fixtures', 'manifest.json') new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') -- cgit v1.2.3 From 6f4e97603c2562914567a85bf18187299c3b543b Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 20:04:28 -0600 Subject: Revert "fix/remove-default-cachebusters" This reverts commit 7410250313f893e5952bb2697324a4d4e3d47d22. This reverts commit cbec33b898efffbfa6acaf91cae45ec0daed4d7a. This reverts commit 345ca3052c395545b90fef9104a16eed5ab051a5, reversing changes made to 47162533af84bb8d26db6d1c9ba1e63d70e9070f. --- CHANGES.txt | 3 +- docs/api/static.rst | 8 +- docs/glossary.rst | 4 - docs/narr/assets.rst | 257 ++++++++++++++++++-------------- pyramid/config/views.py | 20 ++- pyramid/static.py | 178 +++++++++++----------- pyramid/tests/fixtures/manifest.json | 4 - pyramid/tests/fixtures/manifest2.json | 4 - pyramid/tests/test_config/test_views.py | 10 ++ pyramid/tests/test_static.py | 165 ++++++++++++-------- 10 files changed, 368 insertions(+), 285 deletions(-) delete mode 100644 pyramid/tests/fixtures/manifest.json delete mode 100644 pyramid/tests/fixtures/manifest2.json diff --git a/CHANGES.txt b/CHANGES.txt index 18e8ba39c..8b63cf847 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -57,8 +57,7 @@ Features Core APIs are shipped for both cache busting via query strings and path segments and may be extended to fit into custom asset pipelines. See https://github.com/Pylons/pyramid/pull/1380 and - https://github.com/Pylons/pyramid/pull/1583 and - https://github.com/Pylons/pyramid/pull/2013 + https://github.com/Pylons/pyramid/pull/1583 - Add ``pyramid.config.Configurator.root_package`` attribute and init parameter to assist with includeable packages that wish to resolve diff --git a/docs/api/static.rst b/docs/api/static.rst index f3727e197..b6b279139 100644 --- a/docs/api/static.rst +++ b/docs/api/static.rst @@ -9,11 +9,17 @@ :members: :inherited-members: - .. autoclass:: ManifestCacheBuster + .. autoclass:: PathSegmentCacheBuster :members: .. autoclass:: QueryStringCacheBuster :members: + .. autoclass:: PathSegmentMd5CacheBuster + :members: + + .. autoclass:: QueryStringMd5CacheBuster + :members: + .. autoclass:: QueryStringConstantCacheBuster :members: diff --git a/docs/glossary.rst b/docs/glossary.rst index b4bb36421..9c0ea8598 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -1089,7 +1089,3 @@ Glossary data in a Redis database. See https://pypi.python.org/pypi/pyramid_redis_sessions for more information. - cache busting - A technique used when serving a cacheable static asset in order to force - a client to query the new version of the asset. See :ref:`cache_busting` - for more information. diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index d36fa49c0..020794062 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -356,14 +356,14 @@ may change, and then you'll want the client to load a new copy of the asset. Under normal circumstances you'd just need to wait for the client's cached copy to expire before they get the new version of the static resource. -A commonly used workaround to this problem is a technique known as -:term:`cache busting`. Cache busting schemes generally involve generating a -URL for a static asset that changes when the static asset changes. This way -headers can be sent along with the static asset instructing the client to cache -the asset for a very long time. When a static asset is changed, the URL used -to refer to it in a web page also changes, so the client sees it as a new -resource and requests the asset, regardless of any caching policy set for the -resource's old URL. +A commonly used workaround to this problem is a technique known as "cache +busting". Cache busting schemes generally involve generating a URL for a +static asset that changes when the static asset changes. This way headers can +be sent along with the static asset instructing the client to cache the asset +for a very long time. When a static asset is changed, the URL used to refer to +it in a web page also changes, so the client sees it as a new resource and +requests the asset, regardless of any caching policy set for the resource's old +URL. :app:`Pyramid` can be configured to produce cache busting URLs for static assets by passing the optional argument, ``cachebust`` to @@ -372,38 +372,30 @@ assets by passing the optional argument, ``cachebust`` to .. code-block:: python :linenos: - import time - from pyramid.static import QueryStringConstantCacheBuster - # config is an instance of pyramid.config.Configurator - config.add_static_view( - name='static', path='mypackage:folder/static', - cachebust=QueryStringConstantCacheBuster(str(int(time.time()))), - ) + config.add_static_view(name='static', path='mypackage:folder/static', + cachebust=True) Setting the ``cachebust`` argument instructs :app:`Pyramid` to use a cache -busting scheme which adds the curent time for a static asset to the query -string in the asset's URL: +busting scheme which adds the md5 checksum for a static asset as a path segment +in the asset's URL: .. code-block:: python :linenos: js_url = request.static_url('mypackage:folder/static/js/myapp.js') - # Returns: 'http://www.example.com/static/js/myapp.js?x=1445318121' + # Returns: 'http://www.example.com/static/c9658b3c0a314a1ca21e5988e662a09e/js/myapp.js' -When the web server restarts, the time constant will change and therefore so -will its URL. Supplying the ``cachebust`` argument also causes the static -view to set headers instructing clients to cache the asset for ten years, -unless the ``cache_max_age`` argument is also passed, in which case that -value is used. +When the asset changes, so will its md5 checksum, and therefore so will its +URL. Supplying the ``cachebust`` argument also causes the static view to set +headers instructing clients to cache the asset for ten years, unless the +``cache_max_age`` argument is also passed, in which case that value is used. .. note:: - Cache busting is an inherently complex topic as it integrates the asset - pipeline and the web application. It is expected and desired that - application authors will write their own cache buster implementations - conforming to the properties of their own asset pipelines. See - :ref:`custom_cache_busters` for information on writing your own. + md5 checksums are cached in RAM, so if you change a static resource without + restarting your application, you may still generate URLs with a stale md5 + checksum. Disabling the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -414,45 +406,65 @@ configured cache busters without changing calls to ``PYRAMID_PREVENT_CACHEBUST`` environment variable or the ``pyramid.prevent_cachebust`` configuration value to a true value. -.. _custom_cache_busters: - Customizing the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``cachebust`` option to -:meth:`~pyramid.config.Configurator.add_static_view` may be set to any object -that implements the interface :class:`~pyramid.interfaces.ICacheBuster`. +Revisiting from the previous section: + +.. code-block:: python + :linenos: + + # config is an instance of pyramid.config.Configurator + config.add_static_view(name='static', path='mypackage:folder/static', + cachebust=True) + +Setting ``cachebust`` to ``True`` instructs :app:`Pyramid` to use a default +cache busting implementation that should work for many situations. The +``cachebust`` may be set to any object that implements the interface +:class:`~pyramid.interfaces.ICacheBuster`. The above configuration is exactly +equivalent to: + +.. code-block:: python + :linenos: -:app:`Pyramid` ships with a very simplistic + from pyramid.static import PathSegmentMd5CacheBuster + + # config is an instance of pyramid.config.Configurator + config.add_static_view(name='static', path='mypackage:folder/static', + cachebust=PathSegmentMd5CacheBuster()) + +:app:`Pyramid` includes a handful of ready to use cache buster implementations: +:class:`~pyramid.static.PathSegmentMd5CacheBuster`, which inserts an md5 +checksum token in the path portion of the asset's URL, +:class:`~pyramid.static.QueryStringMd5CacheBuster`, which adds an md5 checksum +token to the query string of the asset's URL, and :class:`~pyramid.static.QueryStringConstantCacheBuster`, which adds an -arbitrary token you provide to the query string of the asset's URL. This -is almost never what you want in production as it does not allow fine-grained -busting of individual assets. +arbitrary token you provide to the query string of the asset's URL. In order to implement your own cache buster, you can write your own class from scratch which implements the :class:`~pyramid.interfaces.ICacheBuster` interface. Alternatively you may choose to subclass one of the existing implementations. One of the most likely scenarios is you'd want to change the -way the asset token is generated. To do this just subclass +way the asset token is generated. To do this just subclass either +:class:`~pyramid.static.PathSegmentCacheBuster` or :class:`~pyramid.static.QueryStringCacheBuster` and define a -``tokenize(pathspec)`` method. Here is an example which uses Git to get -the hash of the current commit: +``tokenize(pathspec)`` method. Here is an example which just uses Git to get +the hash of the currently checked out code: .. code-block:: python :linenos: import os import subprocess - from pyramid.static import QueryStringCacheBuster + from pyramid.static import PathSegmentCacheBuster - class GitCacheBuster(QueryStringCacheBuster): + class GitCacheBuster(PathSegmentCacheBuster): """ Assuming your code is installed as a Git checkout, as opposed to an egg from an egg repository like PYPI, you can use this cachebuster to get the current commit's SHA1 to use as the cache bust token. """ - def __init__(self, param='x'): - super(GitCacheBuster, self).__init__(param=param) + def __init__(self): here = os.path.dirname(os.path.abspath(__file__)) self.sha1 = subprocess.check_output( ['git', 'rev-parse', 'HEAD'], @@ -464,60 +476,26 @@ the hash of the current commit: Choosing a Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~ -Many caching HTTP proxies will fail to cache a resource if the URL contains -a query string. Therefore, in general, you should prefer a cache busting -strategy which modifies the path segment rather than methods which add a -token to the query string. - -You will need to consider whether the :app:`Pyramid` application will be -serving your static assets, whether you are using an external asset pipeline -to handle rewriting urls internal to the css/javascript, and how fine-grained -do you want the cache busting tokens to be. - -In many cases you will want to host the static assets on another web server -or externally on a CDN. In these cases your :app:`Pyramid` application may not -even have access to a copy of the static assets. In order to cache bust these -assets you will need some information about them. - -If you are using an external asset pipeline to generate your static files you -should consider using the :class:`~pyramid.static.ManifestCacheBuster`. -This cache buster can load a standard JSON formatted file generated by your -pipeline and use it to cache bust the assets. This has many performance -advantages as :app:`Pyramid` does not need to look at the files to generate -any cache busting tokens, but still supports fine-grained per-file tokens. - -Assuming an example ``manifest.json`` like: - -.. code-block:: json - - { - "css/main.css": "css/main-678b7c80.css", - "images/background.png": "images/background-a8169106.png" - } - -The following code would set up a cachebuster: - -.. code-block:: python - :linenos: - - from pyramid.path import AssetResolver - from pyramid.static import ManifestCacheBuster - - resolver = AssetResolver() - manifest = resolver.resolve('myapp:static/manifest.json') - config.add_static_view( - name='http://mycdn.example.com/', - path='mypackage:static', - cachebust=ManifestCacheBuster(manifest.abspath())) - -A simpler approach is to use the -:class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global -token that will bust all of the assets at once. The advantage of this strategy -is that it is simple and by using the query string there doesn't need to be -any shared information between your application and the static assets. - -The following code would set up a cachebuster that just uses the time at -start up as a cachebust token: +The default cache buster implementation, +:class:`~pyramid.static.PathSegmentMd5CacheBuster`, works very well assuming +that you're using :app:`Pyramid` to serve your static assets. The md5 checksum +is fine grained enough that browsers should only request new versions of +specific assets that have changed. Many caching HTTP proxies will fail to +cache a resource if the URL contains a query string. In general, therefore, +you should prefer a cache busting strategy which modifies the path segment to a +strategy which adds a query string. + +It is possible, however, that your static assets are being served by another +web server or externally on a CDN. In these cases modifying the path segment +for a static asset URL would cause the external service to fail to find the +asset, causing your customer to get a 404. In these cases you would need to +fall back to a cache buster which adds a query string. It is even possible +that there isn't a copy of your static assets available to the :app:`Pyramid` +application, so a cache busting implementation that generates md5 checksums +would fail since it can't access the assets. In such a case, +:class:`~pyramid.static.QueryStringConstantCacheBuster` is a reasonable +fallback. The following code would set up a cachebuster that just uses the +time at start up as a cachebust token: .. code-block:: python :linenos: @@ -528,7 +506,7 @@ start up as a cachebust token: config.add_static_view( name='http://mycdn.example.com/', path='mypackage:static', - cachebust=QueryStringConstantCacheBuster(str(int(time.time())))) + cachebust=QueryStringConstantCacheBuster(str(time.time()))) .. index:: single: static assets view @@ -540,24 +518,85 @@ Often one needs to refer to images and other static assets inside CSS and JavaScript files. If cache busting is active, the final static asset URL is not available until the static assets have been assembled. These URLs cannot be handwritten. Thus, when having static asset references in CSS and JavaScript, -one needs to perform one of the following tasks: +one needs to perform one of the following tasks. * Process the files by using a precompiler which rewrites URLs to their final - cache busted form. Then, you can use the - :class:`~pyramid.static.ManifestCacheBuster` to synchronize your asset - pipeline with :app:`Pyramid`, allowing the pipeline to have full control - over the final URLs of your assets. + cache busted form. * Templatize JS and CSS, and call ``request.static_url()`` inside their template code. * Pass static URL references to CSS and JavaScript via other means. -If your CSS and JavaScript assets use URLs to reference other assets it is -recommended that you implement an external asset pipeline that can rewrite the -generated static files with new URLs containing cache busting tokens. The -machinery inside :app:`Pyramid` will not help with this step as it has very -little knowledge of the asset types your application may use. +Below are some simple approaches for CSS and JS programming which consider +asset cache busting. These approaches do not require additional tools or +packages. + +Relative cache busted URLs in CSS ++++++++++++++++++++++++++++++++++ + +Consider a CSS file ``/static/theme/css/site.css`` which contains the following +CSS code. + +.. code-block:: css + + body { + background: url(/static/theme/img/background.jpg); + } + +Any changes to ``background.jpg`` would not appear to the visitor because the +URL path is not cache busted as it is. Instead we would have to construct an +URL to the background image with the default ``PathSegmentCacheBuster`` cache +busting mechanism:: + + https://site/static/1eeb262c717/theme/img/background.jpg + +Every time the image is updated, the URL would need to be changed. It is not +practical to write this non-human readable URL into a CSS file. + +However, the CSS file itself is cache busted and is located under the path for +static assets. This lets us use relative references in our CSS to cache bust +the image. + +.. code-block:: css + + body { + background: url(../img/background.jpg); + } + +The browser would interpret this as having the CSS file hash in URL:: + + https://site/static/ab234b262c71/theme/css/../img/background.jpg + +The downside of this approach is that if the background image changes, one +needs to bump the CSS file. The CSS file hash change signals the caches that +the relative URL to the image in the CSS has been changed. When updating CSS +and related image assets, updates usually happen hand in hand, so this does not +add extra effort to theming workflow. + +Passing cache busted URLs to JavaScript ++++++++++++++++++++++++++++++++++++++++ + +For JavaScript, one can pass static asset URLs as function arguments or +globals. The globals can be generated in page template code, having access to +the ``request.static_url()`` function. + +Below is a simple example of passing a cached busted image URL in the Jinja2 +template language. Put the following code into the ```` section of the +relevant page. + +.. code-block:: html + + + +Then in your main ``site.js`` file, put the following code. + +.. code-block:: javascript + + var image = new Image(window.assets.backgroundImage); .. _advanced_static: diff --git a/pyramid/config/views.py b/pyramid/config/views.py index e386bc4e1..34fc49c00 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -34,6 +34,7 @@ from pyramid.interfaces import ( ) from pyramid import renderers +from pyramid.static import PathSegmentMd5CacheBuster from pyramid.compat import ( string_types, @@ -1861,12 +1862,14 @@ class ViewsConfiguratorMixin(object): The ``cachebust`` keyword argument may be set to cause :meth:`~pyramid.request.Request.static_url` to use cache busting when generating URLs. See :ref:`cache_busting` for general information - about cache busting. The value of the ``cachebust`` argument must - be an object which implements - :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` - argument is provided, the default for ``cache_max_age`` is modified - to be ten years. ``cache_max_age`` may still be explicitly provided - to override this default. + about cache busting. The value of the ``cachebust`` argument may be + ``True``, in which case a default cache busting implementation is used. + The value of the ``cachebust`` argument may also be an object which + implements :class:`~pyramid.interfaces.ICacheBuster`. See the + :mod:`~pyramid.static` module for some implementations. If the + ``cachebust`` argument is provided, the default for ``cache_max_age`` + is modified to be ten years. ``cache_max_age`` may still be explicitly + provided to override this default. The ``permission`` keyword argument is used to specify the :term:`permission` required by a user to execute the static view. By @@ -1964,6 +1967,9 @@ def isexception(o): @implementer(IStaticURLInfo) class StaticURLInfo(object): + # Indirection for testing + _default_cachebust = PathSegmentMd5CacheBuster + def _get_registrations(self, registry): try: reg = registry._static_url_registrations @@ -2027,6 +2033,8 @@ class StaticURLInfo(object): cb = None else: cb = extra.pop('cachebust', None) + if cb is True: + cb = self._default_cachebust() if cb: def cachebust(subpath, kw): subpath_tuple = tuple(subpath.split('/')) diff --git a/pyramid/static.py b/pyramid/static.py index c2c8c89e5..cb78feb9b 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -1,9 +1,8 @@ # -*- coding: utf-8 -*- -import json +import hashlib import os from os.path import ( - getmtime, normcase, normpath, join, @@ -19,10 +18,7 @@ from pkg_resources import ( from repoze.lru import lru_cache -from pyramid.asset import ( - abspath_from_asset_spec, - resolve_asset_spec, -) +from pyramid.asset import resolve_asset_spec from pyramid.compat import text_ @@ -31,7 +27,7 @@ from pyramid.httpexceptions import ( HTTPMovedPermanently, ) -from pyramid.path import caller_package +from pyramid.path import AssetResolver, caller_package from pyramid.response import FileResponse from pyramid.traversal import traversal_path_info @@ -163,6 +159,71 @@ def _secure_path(path_tuple): encoded = slash.join(path_tuple) # will be unicode return encoded +def _generate_md5(spec): + asset = AssetResolver(None).resolve(spec) + md5 = hashlib.md5() + with asset.stream() as stream: + for block in iter(lambda: stream.read(4096), b''): + md5.update(block) + return md5.hexdigest() + +class Md5AssetTokenGenerator(object): + """ + A mixin class which provides an implementation of + :meth:`~pyramid.interfaces.ICacheBuster.target` which generates an md5 + checksum token for an asset, caching it for subsequent calls. + """ + def __init__(self): + self.token_cache = {} + + def tokenize(self, pathspec): + # An astute observer will notice that this use of token_cache doesn't + # look particularly thread safe. Basic read/write operations on Python + # dicts, however, are atomic, so simply accessing and writing values + # to the dict shouldn't cause a segfault or other catastrophic failure. + # (See: http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm) + # + # We do have a race condition that could result in the same md5 + # checksum getting computed twice or more times in parallel. Since + # the program would still function just fine if this were to occur, + # the extra overhead of using locks to serialize access to the dict + # seems an unnecessary burden. + # + token = self.token_cache.get(pathspec) + if not token: + self.token_cache[pathspec] = token = _generate_md5(pathspec) + return token + +class PathSegmentCacheBuster(object): + """ + An implementation of :class:`~pyramid.interfaces.ICacheBuster` which + inserts a token for cache busting in the path portion of an asset URL. + + To use this class, subclass it and provide a ``tokenize`` method which + accepts a ``pathspec`` and returns a token. + + .. versionadded:: 1.6 + """ + def pregenerate(self, pathspec, subpath, kw): + token = self.tokenize(pathspec) + return (token,) + subpath, kw + + def match(self, subpath): + return subpath[1:] + +class PathSegmentMd5CacheBuster(PathSegmentCacheBuster, + Md5AssetTokenGenerator): + """ + An implementation of :class:`~pyramid.interfaces.ICacheBuster` which + inserts an md5 checksum token for cache busting in the path portion of an + asset URL. Generated md5 checksums are cached in order to speed up + subsequent calls. + + .. versionadded:: 1.6 + """ + def __init__(self): + super(PathSegmentMd5CacheBuster, self).__init__() + class QueryStringCacheBuster(object): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds @@ -188,6 +249,22 @@ class QueryStringCacheBuster(object): kw['_query'] = tuple(query) + ((self.param, token),) return subpath, kw +class QueryStringMd5CacheBuster(QueryStringCacheBuster, + Md5AssetTokenGenerator): + """ + An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds + an md5 checksum token for cache busting in the query string of an asset + URL. Generated md5 checksums are cached in order to speed up subsequent + calls. + + The optional ``param`` argument determines the name of the parameter added + to the query string and defaults to ``'x'``. + + .. versionadded:: 1.6 + """ + def __init__(self, param='x'): + super(QueryStringMd5CacheBuster, self).__init__(param=param) + class QueryStringConstantCacheBuster(QueryStringCacheBuster): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds @@ -207,90 +284,3 @@ class QueryStringConstantCacheBuster(QueryStringCacheBuster): def tokenize(self, pathspec): return self._token - -class ManifestCacheBuster(object): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which - uses a supplied manifest file to map an asset path to a cache-busted - version of the path. - - The ``manifest_spec`` can be an absolute path or a :term:`asset spec` - pointing to a package-relative file. - - The manifest file is expected to conform to the following simple JSON - format: - - .. code-block:: json - - { - "css/main.css": "css/main-678b7c80.css", - "images/background.png": "images/background-a8169106.png", - } - - Specifically, it is a JSON-serialized dictionary where the keys are the - source asset paths used in calls to - :meth:`~pyramid.request.Request.static_url`. For example:: - - .. code-block:: python - - >>> request.static_url('myapp:static/css/main.css') - "http://www.example.com/static/css/main-678b7c80.css" - - If a path is not found in the manifest it will pass through unchanged. - - If ``reload`` is ``True`` then the manifest file will be reloaded when - changed. It is not recommended to leave this enabled in production. - - If the manifest file cannot be found on disk it will be treated as - an empty mapping unless ``reload`` is ``False``. - - The default implementation assumes the requested (possibly cache-busted) - path is the actual filename on disk. Subclasses may override the ``match`` - method to alter this behavior. For example, to strip the cache busting - token from the path. - - .. versionadded:: 1.6 - """ - exists = staticmethod(exists) # testing - getmtime = staticmethod(getmtime) # testing - - def __init__(self, manifest_spec, reload=False): - package_name = caller_package().__name__ - self.manifest_path = abspath_from_asset_spec( - manifest_spec, package_name) - self.reload = reload - - self._mtime = None - if not reload: - self._manifest = self.parse_manifest() - - def parse_manifest(self): - """ - Return a mapping parsed from the ``manifest_path``. - - Subclasses may override this method to use something other than - ``json.loads`` to load any type of file format and return a conforming - dictionary. - - """ - with open(self.manifest_path, 'rb') as fp: - content = fp.read().decode('utf-8') - return json.loads(content) - - @property - def manifest(self): - """ The current manifest dictionary.""" - if self.reload: - if not self.exists(self.manifest_path): - return {} - mtime = self.getmtime(self.manifest_path) - if self._mtime is None or mtime > self._mtime: - self._manifest = self.parse_manifest() - self._mtime = mtime - return self._manifest - - def pregenerate(self, pathspec, subpath, kw): - path = '/'.join(subpath) - path = self.manifest.get(path, path) - new_subpath = path.split('/') - return (new_subpath, kw) diff --git a/pyramid/tests/fixtures/manifest.json b/pyramid/tests/fixtures/manifest.json deleted file mode 100644 index 0a43bc5e3..000000000 --- a/pyramid/tests/fixtures/manifest.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "css/main.css": "css/main-test.css", - "images/background.png": "images/background-a8169106.png" -} diff --git a/pyramid/tests/fixtures/manifest2.json b/pyramid/tests/fixtures/manifest2.json deleted file mode 100644 index fd6b9a7bb..000000000 --- a/pyramid/tests/fixtures/manifest2.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "css/main.css": "css/main-678b7c80.css", - "images/background.png": "images/background-a8169106.png" -} diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index acfb81962..1c2d300a1 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -4104,6 +4104,16 @@ class TestStaticURLInfo(unittest.TestCase): self.assertEqual(config.view_kw['renderer'], 'mypackage:templates/index.pt') + def test_add_cachebust_default(self): + config = self._makeConfig() + inst = self._makeOne() + inst._default_cachebust = lambda: DummyCacheBuster('foo') + inst.add(config, 'view', 'mypackage:path', cachebust=True) + cachebust = config.registry._static_url_registrations[0][3] + subpath, kw = cachebust('some/path', {}) + self.assertEqual(subpath, 'some/path') + self.assertEqual(kw['x'], 'foo') + def test_add_cachebust_prevented(self): config = self._makeConfig() config.registry.settings['pyramid.prevent_cachebust'] = True diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index 4a07c2cb1..a3df74b44 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -1,9 +1,6 @@ import datetime -import os.path import unittest -here = os.path.dirname(__file__) - # 5 years from now (more or less) fiveyrsfuture = datetime.datetime.utcnow() + datetime.timedelta(5*365) @@ -371,14 +368,87 @@ class Test_static_view_use_subpath_True(unittest.TestCase): from pyramid.httpexceptions import HTTPNotFound self.assertRaises(HTTPNotFound, inst, context, request) -class TestQueryStringConstantCacheBuster(unittest.TestCase): +class TestMd5AssetTokenGenerator(unittest.TestCase): + _fspath = None + _tmp = None + + @property + def fspath(self): + if self._fspath: + return self._fspath + + import os + import tempfile + self._tmp = tmp = tempfile.mkdtemp() + self._fspath = os.path.join(tmp, 'test.txt') + return self._fspath + + def tearDown(self): + import shutil + if self._tmp: + shutil.rmtree(self._tmp) + + def _makeOne(self): + from pyramid.static import Md5AssetTokenGenerator as cls + return cls() + + def test_package_resource(self): + fut = self._makeOne().tokenize + expected = '76d653a3a044e2f4b38bb001d283e3d9' + token = fut('pyramid.tests:fixtures/static/index.html') + self.assertEqual(token, expected) + + def test_filesystem_resource(self): + fut = self._makeOne().tokenize + expected = 'd5155f250bef0e9923e894dbc713c5dd' + with open(self.fspath, 'w') as f: + f.write("Are we rich yet?") + token = fut(self.fspath) + self.assertEqual(token, expected) + + def test_cache(self): + fut = self._makeOne().tokenize + expected = 'd5155f250bef0e9923e894dbc713c5dd' + with open(self.fspath, 'w') as f: + f.write("Are we rich yet?") + token = fut(self.fspath) + self.assertEqual(token, expected) + + # md5 shouldn't change because we've cached it + with open(self.fspath, 'w') as f: + f.write("Sorry for the convenience.") + token = fut(self.fspath) + self.assertEqual(token, expected) + +class TestPathSegmentMd5CacheBuster(unittest.TestCase): + + def _makeOne(self): + from pyramid.static import PathSegmentMd5CacheBuster as cls + inst = cls() + inst.tokenize = lambda pathspec: 'foo' + return inst + + def test_token(self): + fut = self._makeOne().tokenize + self.assertEqual(fut('whatever'), 'foo') + + def test_pregenerate(self): + fut = self._makeOne().pregenerate + self.assertEqual(fut('foo', ('bar',), 'kw'), (('foo', 'bar'), 'kw')) + + def test_match(self): + fut = self._makeOne().match + self.assertEqual(fut(('foo', 'bar')), ('bar',)) + +class TestQueryStringMd5CacheBuster(unittest.TestCase): def _makeOne(self, param=None): - from pyramid.static import QueryStringConstantCacheBuster as cls + from pyramid.static import QueryStringMd5CacheBuster as cls if param: - inst = cls('foo', param) + inst = cls(param) else: - inst = cls('foo') + inst = cls() + inst.tokenize = lambda pathspec: 'foo' return inst def test_token(self): @@ -409,70 +479,43 @@ class TestQueryStringConstantCacheBuster(unittest.TestCase): fut('foo', ('bar',), {'_query': (('a', 'b'),)}), (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) -class TestManifestCacheBuster(unittest.TestCase): - - def _makeOne(self, path, **kw): - from pyramid.static import ManifestCacheBuster as cls - return cls(path, **kw) - - def test_it(self): - manifest_path = os.path.join(here, 'fixtures', 'manifest.json') - fut = self._makeOne(manifest_path).pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) - self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) - - def test_it_with_relspec(self): - fut = self._makeOne('fixtures/manifest.json').pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) - self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) +class TestQueryStringConstantCacheBuster(TestQueryStringMd5CacheBuster): - def test_it_with_absspec(self): - fut = self._makeOne('pyramid.tests:fixtures/manifest.json').pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) - self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + def _makeOne(self, param=None): + from pyramid.static import QueryStringConstantCacheBuster as cls + if param: + inst = cls('foo', param) + else: + inst = cls('foo') + return inst - def test_reload(self): - manifest_path = os.path.join(here, 'fixtures', 'manifest.json') - new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') - inst = self._makeOne('foo', reload=True) - inst.getmtime = lambda *args, **kwargs: 0 - fut = inst.pregenerate + def test_token(self): + fut = self._makeOne().tokenize + self.assertEqual(fut('whatever'), 'foo') - # test without a valid manifest + def test_pregenerate(self): + fut = self._makeOne().pregenerate self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main.css'], {})) + fut('foo', ('bar',), {}), + (('bar',), {'_query': {'x': 'foo'}})) - # swap to a real manifest, setting mtime to 0 - inst.manifest_path = manifest_path + def test_pregenerate_change_param(self): + fut = self._makeOne('y').pregenerate self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', ('bar',), {}), + (('bar',), {'_query': {'y': 'foo'}})) - # ensure switching the path doesn't change the result - inst.manifest_path = new_manifest_path + def test_pregenerate_query_is_already_tuples(self): + fut = self._makeOne().pregenerate self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', ('bar',), {'_query': [('a', 'b')]}), + (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) - # update mtime, should cause a reload - inst.getmtime = lambda *args, **kwargs: 1 + def test_pregenerate_query_is_tuple_of_tuples(self): + fut = self._makeOne().pregenerate self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-678b7c80.css'], {})) - - def test_invalid_manifest(self): - self.assertRaises(IOError, lambda: self._makeOne('foo')) - - def test_invalid_manifest_with_reload(self): - inst = self._makeOne('foo', reload=True) - self.assertEqual(inst.manifest, {}) + fut('foo', ('bar',), {'_query': (('a', 'b'),)}), + (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) class DummyContext: pass -- cgit v1.2.3 From 3a41196208c7fd78cfb177bb5105c6a702436b78 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 20 Oct 2015 00:32:14 -0500 Subject: update cache buster prose and add ManifestCacheBuster redux of #2013 --- docs/api/static.rst | 8 +- docs/glossary.rst | 4 + docs/narr/assets.rst | 257 ++++++++++++++------------------ pyramid/config/views.py | 20 +-- pyramid/static.py | 166 ++++++++++----------- pyramid/tests/fixtures/manifest.json | 4 + pyramid/tests/fixtures/manifest2.json | 4 + pyramid/tests/test_config/test_views.py | 10 -- pyramid/tests/test_static.py | 151 ++++++------------- 9 files changed, 258 insertions(+), 366 deletions(-) create mode 100644 pyramid/tests/fixtures/manifest.json create mode 100644 pyramid/tests/fixtures/manifest2.json diff --git a/docs/api/static.rst b/docs/api/static.rst index b6b279139..f3727e197 100644 --- a/docs/api/static.rst +++ b/docs/api/static.rst @@ -9,17 +9,11 @@ :members: :inherited-members: - .. autoclass:: PathSegmentCacheBuster + .. autoclass:: ManifestCacheBuster :members: .. autoclass:: QueryStringCacheBuster :members: - .. autoclass:: PathSegmentMd5CacheBuster - :members: - - .. autoclass:: QueryStringMd5CacheBuster - :members: - .. autoclass:: QueryStringConstantCacheBuster :members: diff --git a/docs/glossary.rst b/docs/glossary.rst index 9c0ea8598..b4bb36421 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -1089,3 +1089,7 @@ Glossary data in a Redis database. See https://pypi.python.org/pypi/pyramid_redis_sessions for more information. + cache busting + A technique used when serving a cacheable static asset in order to force + a client to query the new version of the asset. See :ref:`cache_busting` + for more information. diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 020794062..d36fa49c0 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -356,14 +356,14 @@ may change, and then you'll want the client to load a new copy of the asset. Under normal circumstances you'd just need to wait for the client's cached copy to expire before they get the new version of the static resource. -A commonly used workaround to this problem is a technique known as "cache -busting". Cache busting schemes generally involve generating a URL for a -static asset that changes when the static asset changes. This way headers can -be sent along with the static asset instructing the client to cache the asset -for a very long time. When a static asset is changed, the URL used to refer to -it in a web page also changes, so the client sees it as a new resource and -requests the asset, regardless of any caching policy set for the resource's old -URL. +A commonly used workaround to this problem is a technique known as +:term:`cache busting`. Cache busting schemes generally involve generating a +URL for a static asset that changes when the static asset changes. This way +headers can be sent along with the static asset instructing the client to cache +the asset for a very long time. When a static asset is changed, the URL used +to refer to it in a web page also changes, so the client sees it as a new +resource and requests the asset, regardless of any caching policy set for the +resource's old URL. :app:`Pyramid` can be configured to produce cache busting URLs for static assets by passing the optional argument, ``cachebust`` to @@ -372,30 +372,38 @@ assets by passing the optional argument, ``cachebust`` to .. code-block:: python :linenos: + import time + from pyramid.static import QueryStringConstantCacheBuster + # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=True) + config.add_static_view( + name='static', path='mypackage:folder/static', + cachebust=QueryStringConstantCacheBuster(str(int(time.time()))), + ) Setting the ``cachebust`` argument instructs :app:`Pyramid` to use a cache -busting scheme which adds the md5 checksum for a static asset as a path segment -in the asset's URL: +busting scheme which adds the curent time for a static asset to the query +string in the asset's URL: .. code-block:: python :linenos: js_url = request.static_url('mypackage:folder/static/js/myapp.js') - # Returns: 'http://www.example.com/static/c9658b3c0a314a1ca21e5988e662a09e/js/myapp.js' + # Returns: 'http://www.example.com/static/js/myapp.js?x=1445318121' -When the asset changes, so will its md5 checksum, and therefore so will its -URL. Supplying the ``cachebust`` argument also causes the static view to set -headers instructing clients to cache the asset for ten years, unless the -``cache_max_age`` argument is also passed, in which case that value is used. +When the web server restarts, the time constant will change and therefore so +will its URL. Supplying the ``cachebust`` argument also causes the static +view to set headers instructing clients to cache the asset for ten years, +unless the ``cache_max_age`` argument is also passed, in which case that +value is used. .. note:: - md5 checksums are cached in RAM, so if you change a static resource without - restarting your application, you may still generate URLs with a stale md5 - checksum. + Cache busting is an inherently complex topic as it integrates the asset + pipeline and the web application. It is expected and desired that + application authors will write their own cache buster implementations + conforming to the properties of their own asset pipelines. See + :ref:`custom_cache_busters` for information on writing your own. Disabling the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -406,65 +414,45 @@ configured cache busters without changing calls to ``PYRAMID_PREVENT_CACHEBUST`` environment variable or the ``pyramid.prevent_cachebust`` configuration value to a true value. +.. _custom_cache_busters: + Customizing the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Revisiting from the previous section: - -.. code-block:: python - :linenos: - - # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=True) - -Setting ``cachebust`` to ``True`` instructs :app:`Pyramid` to use a default -cache busting implementation that should work for many situations. The -``cachebust`` may be set to any object that implements the interface -:class:`~pyramid.interfaces.ICacheBuster`. The above configuration is exactly -equivalent to: - -.. code-block:: python - :linenos: +The ``cachebust`` option to +:meth:`~pyramid.config.Configurator.add_static_view` may be set to any object +that implements the interface :class:`~pyramid.interfaces.ICacheBuster`. - from pyramid.static import PathSegmentMd5CacheBuster - - # config is an instance of pyramid.config.Configurator - config.add_static_view(name='static', path='mypackage:folder/static', - cachebust=PathSegmentMd5CacheBuster()) - -:app:`Pyramid` includes a handful of ready to use cache buster implementations: -:class:`~pyramid.static.PathSegmentMd5CacheBuster`, which inserts an md5 -checksum token in the path portion of the asset's URL, -:class:`~pyramid.static.QueryStringMd5CacheBuster`, which adds an md5 checksum -token to the query string of the asset's URL, and +:app:`Pyramid` ships with a very simplistic :class:`~pyramid.static.QueryStringConstantCacheBuster`, which adds an -arbitrary token you provide to the query string of the asset's URL. +arbitrary token you provide to the query string of the asset's URL. This +is almost never what you want in production as it does not allow fine-grained +busting of individual assets. In order to implement your own cache buster, you can write your own class from scratch which implements the :class:`~pyramid.interfaces.ICacheBuster` interface. Alternatively you may choose to subclass one of the existing implementations. One of the most likely scenarios is you'd want to change the -way the asset token is generated. To do this just subclass either -:class:`~pyramid.static.PathSegmentCacheBuster` or +way the asset token is generated. To do this just subclass :class:`~pyramid.static.QueryStringCacheBuster` and define a -``tokenize(pathspec)`` method. Here is an example which just uses Git to get -the hash of the currently checked out code: +``tokenize(pathspec)`` method. Here is an example which uses Git to get +the hash of the current commit: .. code-block:: python :linenos: import os import subprocess - from pyramid.static import PathSegmentCacheBuster + from pyramid.static import QueryStringCacheBuster - class GitCacheBuster(PathSegmentCacheBuster): + class GitCacheBuster(QueryStringCacheBuster): """ Assuming your code is installed as a Git checkout, as opposed to an egg from an egg repository like PYPI, you can use this cachebuster to get the current commit's SHA1 to use as the cache bust token. """ - def __init__(self): + def __init__(self, param='x'): + super(GitCacheBuster, self).__init__(param=param) here = os.path.dirname(os.path.abspath(__file__)) self.sha1 = subprocess.check_output( ['git', 'rev-parse', 'HEAD'], @@ -476,26 +464,60 @@ the hash of the currently checked out code: Choosing a Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~ -The default cache buster implementation, -:class:`~pyramid.static.PathSegmentMd5CacheBuster`, works very well assuming -that you're using :app:`Pyramid` to serve your static assets. The md5 checksum -is fine grained enough that browsers should only request new versions of -specific assets that have changed. Many caching HTTP proxies will fail to -cache a resource if the URL contains a query string. In general, therefore, -you should prefer a cache busting strategy which modifies the path segment to a -strategy which adds a query string. - -It is possible, however, that your static assets are being served by another -web server or externally on a CDN. In these cases modifying the path segment -for a static asset URL would cause the external service to fail to find the -asset, causing your customer to get a 404. In these cases you would need to -fall back to a cache buster which adds a query string. It is even possible -that there isn't a copy of your static assets available to the :app:`Pyramid` -application, so a cache busting implementation that generates md5 checksums -would fail since it can't access the assets. In such a case, -:class:`~pyramid.static.QueryStringConstantCacheBuster` is a reasonable -fallback. The following code would set up a cachebuster that just uses the -time at start up as a cachebust token: +Many caching HTTP proxies will fail to cache a resource if the URL contains +a query string. Therefore, in general, you should prefer a cache busting +strategy which modifies the path segment rather than methods which add a +token to the query string. + +You will need to consider whether the :app:`Pyramid` application will be +serving your static assets, whether you are using an external asset pipeline +to handle rewriting urls internal to the css/javascript, and how fine-grained +do you want the cache busting tokens to be. + +In many cases you will want to host the static assets on another web server +or externally on a CDN. In these cases your :app:`Pyramid` application may not +even have access to a copy of the static assets. In order to cache bust these +assets you will need some information about them. + +If you are using an external asset pipeline to generate your static files you +should consider using the :class:`~pyramid.static.ManifestCacheBuster`. +This cache buster can load a standard JSON formatted file generated by your +pipeline and use it to cache bust the assets. This has many performance +advantages as :app:`Pyramid` does not need to look at the files to generate +any cache busting tokens, but still supports fine-grained per-file tokens. + +Assuming an example ``manifest.json`` like: + +.. code-block:: json + + { + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png" + } + +The following code would set up a cachebuster: + +.. code-block:: python + :linenos: + + from pyramid.path import AssetResolver + from pyramid.static import ManifestCacheBuster + + resolver = AssetResolver() + manifest = resolver.resolve('myapp:static/manifest.json') + config.add_static_view( + name='http://mycdn.example.com/', + path='mypackage:static', + cachebust=ManifestCacheBuster(manifest.abspath())) + +A simpler approach is to use the +:class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global +token that will bust all of the assets at once. The advantage of this strategy +is that it is simple and by using the query string there doesn't need to be +any shared information between your application and the static assets. + +The following code would set up a cachebuster that just uses the time at +start up as a cachebust token: .. code-block:: python :linenos: @@ -506,7 +528,7 @@ time at start up as a cachebust token: config.add_static_view( name='http://mycdn.example.com/', path='mypackage:static', - cachebust=QueryStringConstantCacheBuster(str(time.time()))) + cachebust=QueryStringConstantCacheBuster(str(int(time.time())))) .. index:: single: static assets view @@ -518,85 +540,24 @@ Often one needs to refer to images and other static assets inside CSS and JavaScript files. If cache busting is active, the final static asset URL is not available until the static assets have been assembled. These URLs cannot be handwritten. Thus, when having static asset references in CSS and JavaScript, -one needs to perform one of the following tasks. +one needs to perform one of the following tasks: * Process the files by using a precompiler which rewrites URLs to their final - cache busted form. + cache busted form. Then, you can use the + :class:`~pyramid.static.ManifestCacheBuster` to synchronize your asset + pipeline with :app:`Pyramid`, allowing the pipeline to have full control + over the final URLs of your assets. * Templatize JS and CSS, and call ``request.static_url()`` inside their template code. * Pass static URL references to CSS and JavaScript via other means. -Below are some simple approaches for CSS and JS programming which consider -asset cache busting. These approaches do not require additional tools or -packages. - -Relative cache busted URLs in CSS -+++++++++++++++++++++++++++++++++ - -Consider a CSS file ``/static/theme/css/site.css`` which contains the following -CSS code. - -.. code-block:: css - - body { - background: url(/static/theme/img/background.jpg); - } - -Any changes to ``background.jpg`` would not appear to the visitor because the -URL path is not cache busted as it is. Instead we would have to construct an -URL to the background image with the default ``PathSegmentCacheBuster`` cache -busting mechanism:: - - https://site/static/1eeb262c717/theme/img/background.jpg - -Every time the image is updated, the URL would need to be changed. It is not -practical to write this non-human readable URL into a CSS file. - -However, the CSS file itself is cache busted and is located under the path for -static assets. This lets us use relative references in our CSS to cache bust -the image. - -.. code-block:: css - - body { - background: url(../img/background.jpg); - } - -The browser would interpret this as having the CSS file hash in URL:: - - https://site/static/ab234b262c71/theme/css/../img/background.jpg - -The downside of this approach is that if the background image changes, one -needs to bump the CSS file. The CSS file hash change signals the caches that -the relative URL to the image in the CSS has been changed. When updating CSS -and related image assets, updates usually happen hand in hand, so this does not -add extra effort to theming workflow. - -Passing cache busted URLs to JavaScript -+++++++++++++++++++++++++++++++++++++++ - -For JavaScript, one can pass static asset URLs as function arguments or -globals. The globals can be generated in page template code, having access to -the ``request.static_url()`` function. - -Below is a simple example of passing a cached busted image URL in the Jinja2 -template language. Put the following code into the ```` section of the -relevant page. - -.. code-block:: html - - - -Then in your main ``site.js`` file, put the following code. - -.. code-block:: javascript - - var image = new Image(window.assets.backgroundImage); +If your CSS and JavaScript assets use URLs to reference other assets it is +recommended that you implement an external asset pipeline that can rewrite the +generated static files with new URLs containing cache busting tokens. The +machinery inside :app:`Pyramid` will not help with this step as it has very +little knowledge of the asset types your application may use. .. _advanced_static: diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 34fc49c00..e386bc4e1 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -34,7 +34,6 @@ from pyramid.interfaces import ( ) from pyramid import renderers -from pyramid.static import PathSegmentMd5CacheBuster from pyramid.compat import ( string_types, @@ -1862,14 +1861,12 @@ class ViewsConfiguratorMixin(object): The ``cachebust`` keyword argument may be set to cause :meth:`~pyramid.request.Request.static_url` to use cache busting when generating URLs. See :ref:`cache_busting` for general information - about cache busting. The value of the ``cachebust`` argument may be - ``True``, in which case a default cache busting implementation is used. - The value of the ``cachebust`` argument may also be an object which - implements :class:`~pyramid.interfaces.ICacheBuster`. See the - :mod:`~pyramid.static` module for some implementations. If the - ``cachebust`` argument is provided, the default for ``cache_max_age`` - is modified to be ten years. ``cache_max_age`` may still be explicitly - provided to override this default. + about cache busting. The value of the ``cachebust`` argument must + be an object which implements + :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` + argument is provided, the default for ``cache_max_age`` is modified + to be ten years. ``cache_max_age`` may still be explicitly provided + to override this default. The ``permission`` keyword argument is used to specify the :term:`permission` required by a user to execute the static view. By @@ -1967,9 +1964,6 @@ def isexception(o): @implementer(IStaticURLInfo) class StaticURLInfo(object): - # Indirection for testing - _default_cachebust = PathSegmentMd5CacheBuster - def _get_registrations(self, registry): try: reg = registry._static_url_registrations @@ -2033,8 +2027,6 @@ class StaticURLInfo(object): cb = None else: cb = extra.pop('cachebust', None) - if cb is True: - cb = self._default_cachebust() if cb: def cachebust(subpath, kw): subpath_tuple = tuple(subpath.split('/')) diff --git a/pyramid/static.py b/pyramid/static.py index cb78feb9b..59f440c82 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -1,8 +1,9 @@ # -*- coding: utf-8 -*- -import hashlib +import json import os from os.path import ( + getmtime, normcase, normpath, join, @@ -27,7 +28,7 @@ from pyramid.httpexceptions import ( HTTPMovedPermanently, ) -from pyramid.path import AssetResolver, caller_package +from pyramid.path import caller_package from pyramid.response import FileResponse from pyramid.traversal import traversal_path_info @@ -159,71 +160,6 @@ def _secure_path(path_tuple): encoded = slash.join(path_tuple) # will be unicode return encoded -def _generate_md5(spec): - asset = AssetResolver(None).resolve(spec) - md5 = hashlib.md5() - with asset.stream() as stream: - for block in iter(lambda: stream.read(4096), b''): - md5.update(block) - return md5.hexdigest() - -class Md5AssetTokenGenerator(object): - """ - A mixin class which provides an implementation of - :meth:`~pyramid.interfaces.ICacheBuster.target` which generates an md5 - checksum token for an asset, caching it for subsequent calls. - """ - def __init__(self): - self.token_cache = {} - - def tokenize(self, pathspec): - # An astute observer will notice that this use of token_cache doesn't - # look particularly thread safe. Basic read/write operations on Python - # dicts, however, are atomic, so simply accessing and writing values - # to the dict shouldn't cause a segfault or other catastrophic failure. - # (See: http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm) - # - # We do have a race condition that could result in the same md5 - # checksum getting computed twice or more times in parallel. Since - # the program would still function just fine if this were to occur, - # the extra overhead of using locks to serialize access to the dict - # seems an unnecessary burden. - # - token = self.token_cache.get(pathspec) - if not token: - self.token_cache[pathspec] = token = _generate_md5(pathspec) - return token - -class PathSegmentCacheBuster(object): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which - inserts a token for cache busting in the path portion of an asset URL. - - To use this class, subclass it and provide a ``tokenize`` method which - accepts a ``pathspec`` and returns a token. - - .. versionadded:: 1.6 - """ - def pregenerate(self, pathspec, subpath, kw): - token = self.tokenize(pathspec) - return (token,) + subpath, kw - - def match(self, subpath): - return subpath[1:] - -class PathSegmentMd5CacheBuster(PathSegmentCacheBuster, - Md5AssetTokenGenerator): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which - inserts an md5 checksum token for cache busting in the path portion of an - asset URL. Generated md5 checksums are cached in order to speed up - subsequent calls. - - .. versionadded:: 1.6 - """ - def __init__(self): - super(PathSegmentMd5CacheBuster, self).__init__() - class QueryStringCacheBuster(object): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds @@ -249,22 +185,6 @@ class QueryStringCacheBuster(object): kw['_query'] = tuple(query) + ((self.param, token),) return subpath, kw -class QueryStringMd5CacheBuster(QueryStringCacheBuster, - Md5AssetTokenGenerator): - """ - An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds - an md5 checksum token for cache busting in the query string of an asset - URL. Generated md5 checksums are cached in order to speed up subsequent - calls. - - The optional ``param`` argument determines the name of the parameter added - to the query string and defaults to ``'x'``. - - .. versionadded:: 1.6 - """ - def __init__(self, param='x'): - super(QueryStringMd5CacheBuster, self).__init__(param=param) - class QueryStringConstantCacheBuster(QueryStringCacheBuster): """ An implementation of :class:`~pyramid.interfaces.ICacheBuster` which adds @@ -284,3 +204,83 @@ class QueryStringConstantCacheBuster(QueryStringCacheBuster): def tokenize(self, pathspec): return self._token + +class ManifestCacheBuster(object): + """ + An implementation of :class:`~pyramid.interfaces.ICacheBuster` which + uses a supplied manifest file to map an asset path to a cache-busted + version of the path. + + The file is expected to conform to the following simple JSON format: + + .. code-block:: json + + { + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png", + } + + Specifically, it is a JSON-serialized dictionary where the keys are the + source asset paths used in calls to + :meth:`~pyramid.request.Request.static_url. For example:: + + .. code-block:: python + + >>> request.static_url('myapp:static/css/main.css') + "http://www.example.com/static/css/main-678b7c80.css" + + If a path is not found in the manifest it will pass through unchanged. + + If ``reload`` is ``True`` then the manifest file will be reloaded when + changed. It is not recommended to leave this enabled in production. + + If the manifest file cannot be found on disk it will be treated as + an empty mapping unless ``reload`` is ``False``. + + The default implementation assumes the requested (possibly cache-busted) + path is the actual filename on disk. Subclasses may override the ``match`` + method to alter this behavior. For example, to strip the cache busting + token from the path. + + .. versionadded:: 1.6 + """ + exists = staticmethod(exists) # testing + getmtime = staticmethod(getmtime) # testing + + def __init__(self, manifest_path, reload=False): + self.manifest_path = manifest_path + self.reload = reload + + self._mtime = None + if not reload: + self._manifest = self.parse_manifest() + + def parse_manifest(self): + """ + Return a mapping parsed from the ``manifest_path``. + + Subclasses may override this method to use something other than + ``json.loads``. + + """ + with open(self.manifest_path, 'rb') as fp: + content = fp.read().decode('utf-8') + return json.loads(content) + + @property + def manifest(self): + """ The current manifest dictionary.""" + if self.reload: + if not self.exists(self.manifest_path): + return {} + mtime = self.getmtime(self.manifest_path) + if self._mtime is None or mtime > self._mtime: + self._manifest = self.parse_manifest() + self._mtime = mtime + return self._manifest + + def pregenerate(self, pathspec, subpath, kw): + path = '/'.join(subpath) + path = self.manifest.get(path, path) + new_subpath = path.split('/') + return (new_subpath, kw) diff --git a/pyramid/tests/fixtures/manifest.json b/pyramid/tests/fixtures/manifest.json new file mode 100644 index 000000000..0a43bc5e3 --- /dev/null +++ b/pyramid/tests/fixtures/manifest.json @@ -0,0 +1,4 @@ +{ + "css/main.css": "css/main-test.css", + "images/background.png": "images/background-a8169106.png" +} diff --git a/pyramid/tests/fixtures/manifest2.json b/pyramid/tests/fixtures/manifest2.json new file mode 100644 index 000000000..fd6b9a7bb --- /dev/null +++ b/pyramid/tests/fixtures/manifest2.json @@ -0,0 +1,4 @@ +{ + "css/main.css": "css/main-678b7c80.css", + "images/background.png": "images/background-a8169106.png" +} diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index 1c2d300a1..acfb81962 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -4104,16 +4104,6 @@ class TestStaticURLInfo(unittest.TestCase): self.assertEqual(config.view_kw['renderer'], 'mypackage:templates/index.pt') - def test_add_cachebust_default(self): - config = self._makeConfig() - inst = self._makeOne() - inst._default_cachebust = lambda: DummyCacheBuster('foo') - inst.add(config, 'view', 'mypackage:path', cachebust=True) - cachebust = config.registry._static_url_registrations[0][3] - subpath, kw = cachebust('some/path', {}) - self.assertEqual(subpath, 'some/path') - self.assertEqual(kw['x'], 'foo') - def test_add_cachebust_prevented(self): config = self._makeConfig() config.registry.settings['pyramid.prevent_cachebust'] = True diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index a3df74b44..ac30e9e50 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -1,6 +1,9 @@ import datetime +import os.path import unittest +here = os.path.dirname(__file__) + # 5 years from now (more or less) fiveyrsfuture = datetime.datetime.utcnow() + datetime.timedelta(5*365) @@ -368,87 +371,14 @@ class Test_static_view_use_subpath_True(unittest.TestCase): from pyramid.httpexceptions import HTTPNotFound self.assertRaises(HTTPNotFound, inst, context, request) -class TestMd5AssetTokenGenerator(unittest.TestCase): - _fspath = None - _tmp = None - - @property - def fspath(self): - if self._fspath: - return self._fspath - - import os - import tempfile - self._tmp = tmp = tempfile.mkdtemp() - self._fspath = os.path.join(tmp, 'test.txt') - return self._fspath - - def tearDown(self): - import shutil - if self._tmp: - shutil.rmtree(self._tmp) - - def _makeOne(self): - from pyramid.static import Md5AssetTokenGenerator as cls - return cls() - - def test_package_resource(self): - fut = self._makeOne().tokenize - expected = '76d653a3a044e2f4b38bb001d283e3d9' - token = fut('pyramid.tests:fixtures/static/index.html') - self.assertEqual(token, expected) - - def test_filesystem_resource(self): - fut = self._makeOne().tokenize - expected = 'd5155f250bef0e9923e894dbc713c5dd' - with open(self.fspath, 'w') as f: - f.write("Are we rich yet?") - token = fut(self.fspath) - self.assertEqual(token, expected) - - def test_cache(self): - fut = self._makeOne().tokenize - expected = 'd5155f250bef0e9923e894dbc713c5dd' - with open(self.fspath, 'w') as f: - f.write("Are we rich yet?") - token = fut(self.fspath) - self.assertEqual(token, expected) - - # md5 shouldn't change because we've cached it - with open(self.fspath, 'w') as f: - f.write("Sorry for the convenience.") - token = fut(self.fspath) - self.assertEqual(token, expected) - -class TestPathSegmentMd5CacheBuster(unittest.TestCase): - - def _makeOne(self): - from pyramid.static import PathSegmentMd5CacheBuster as cls - inst = cls() - inst.tokenize = lambda pathspec: 'foo' - return inst - - def test_token(self): - fut = self._makeOne().tokenize - self.assertEqual(fut('whatever'), 'foo') - - def test_pregenerate(self): - fut = self._makeOne().pregenerate - self.assertEqual(fut('foo', ('bar',), 'kw'), (('foo', 'bar'), 'kw')) - - def test_match(self): - fut = self._makeOne().match - self.assertEqual(fut(('foo', 'bar')), ('bar',)) - -class TestQueryStringMd5CacheBuster(unittest.TestCase): +class TestQueryStringConstantCacheBuster(unittest.TestCase): def _makeOne(self, param=None): - from pyramid.static import QueryStringMd5CacheBuster as cls + from pyramid.static import QueryStringConstantCacheBuster as cls if param: - inst = cls(param) + inst = cls('foo', param) else: - inst = cls() - inst.tokenize = lambda pathspec: 'foo' + inst = cls('foo') return inst def test_token(self): @@ -479,43 +409,56 @@ class TestQueryStringMd5CacheBuster(unittest.TestCase): fut('foo', ('bar',), {'_query': (('a', 'b'),)}), (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) -class TestQueryStringConstantCacheBuster(TestQueryStringMd5CacheBuster): +class TestManifestCacheBuster(unittest.TestCase): - def _makeOne(self, param=None): - from pyramid.static import QueryStringConstantCacheBuster as cls - if param: - inst = cls('foo', param) - else: - inst = cls('foo') - return inst + def _makeOne(self, path, **kw): + from pyramid.static import ManifestCacheBuster as cls + return cls(path, **kw) - def test_token(self): - fut = self._makeOne().tokenize - self.assertEqual(fut('whatever'), 'foo') + def test_it(self): + manifest_path = os.path.join(here, 'fixtures', 'manifest.json') + fut = self._makeOne(manifest_path).pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) - def test_pregenerate(self): - fut = self._makeOne().pregenerate + def test_reload(self): + manifest_path = os.path.join(here, 'fixtures', 'manifest.json') + new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') + inst = self._makeOne('foo', reload=True) + inst.getmtime = lambda *args, **kwargs: 0 + fut = inst.pregenerate + + # test without a valid manifest self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'x': 'foo'}})) + fut('foo', ('css', 'main.css'), {}), + (['css', 'main.css'], {})) - def test_pregenerate_change_param(self): - fut = self._makeOne('y').pregenerate + # swap to a real manifest, setting mtime to 0 + inst.manifest_path = manifest_path self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'y': 'foo'}})) + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) - def test_pregenerate_query_is_already_tuples(self): - fut = self._makeOne().pregenerate + # ensure switching the path doesn't change the result + inst.manifest_path = new_manifest_path self.assertEqual( - fut('foo', ('bar',), {'_query': [('a', 'b')]}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) - def test_pregenerate_query_is_tuple_of_tuples(self): - fut = self._makeOne().pregenerate + # update mtime, should cause a reload + inst.getmtime = lambda *args, **kwargs: 1 self.assertEqual( - fut('foo', ('bar',), {'_query': (('a', 'b'),)}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-678b7c80.css'], {})) + + def test_invalid_manifest(self): + self.assertRaises(IOError, lambda: self._makeOne('foo')) + + def test_invalid_manifest_with_reload(self): + inst = self._makeOne('foo', reload=True) + self.assertEqual(inst.manifest, {}) class DummyContext: pass -- cgit v1.2.3 From b084d76c8941609caca5b8fcd451ce516c1b3611 Mon Sep 17 00:00:00 2001 From: Bert JW Regeer Date: Thu, 12 Nov 2015 21:50:15 -0700 Subject: Simplify tests --- pyramid/tests/test_authentication.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/pyramid/tests/test_authentication.py b/pyramid/tests/test_authentication.py index 1a367fd15..4a6525af2 100644 --- a/pyramid/tests/test_authentication.py +++ b/pyramid/tests/test_authentication.py @@ -761,17 +761,13 @@ class TestAuthTktCookieHelper(unittest.TestCase): result = helper.identify(request) self.assertEqual(result, None) - def test_identify_cookie_timed_out(self): + def test_identify_cookie_timeout(self): helper = self._makeOne('secret', timeout=1) - request = self._makeRequest({'HTTP_COOKIE':'auth_tkt=bogus'}) - result = helper.identify(request) - self.assertEqual(result, None) + self.assertEqual(helper.timeout, 1) def test_identify_cookie_str_timeout(self): helper = self._makeOne('secret', timeout='1') - request = self._makeRequest({'HTTP_COOKIE':'auth_tkt=bogus'}) - result = helper.identify(request) - self.assertEqual(result, None) + self.assertEqual(helper.timeout, 1) def test_identify_cookie_reissue(self): import time -- cgit v1.2.3 From d9f7eddd63ff8b8ec6de3498bd33b234af0a92e5 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 22:52:29 -0600 Subject: fix usage of --monitor-restart with --daemon Fixes #1216 --- pyramid/scripts/pserve.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 1bfedf384..63f34f6c2 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -302,7 +302,10 @@ class PServeCommand(object): raise ValueError(msg) writeable_pid_file.close() - if getattr(self.options, 'daemon', False): + if ( + getattr(self.options, 'daemon', False) and + not os.environ.get(self._monitor_environ_key) + ): self._warn_daemon_deprecated() try: self.daemonize() @@ -311,15 +314,18 @@ class PServeCommand(object): self.out(str(ex)) return 2 + if ( + not os.environ.get(self._monitor_environ_key) and + self.options.pid_file + ): + self.record_pid(self.options.pid_file) + if ( self.options.monitor_restart and not os.environ.get(self._monitor_environ_key) ): return self.restart_with_monitor() - if self.options.pid_file: - self.record_pid(self.options.pid_file) - if self.options.log_file: stdout_log = LazyWriter(self.options.log_file, 'a') sys.stdout = stdout_log -- cgit v1.2.3 From 389b450da27a91fb413a5eab5ed4a704537ac105 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 22:58:46 -0600 Subject: add changelog entry for #2118 --- CHANGES.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 8b63cf847..77b8e9298 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -232,6 +232,9 @@ Bug Fixes shell a little more straightfoward. See https://github.com/Pylons/pyramid/pull/1883 +- Fixed usage of ``pserve --monitor-restart --daemon`` which would fail in + horrible ways. See https://github.com/Pylons/pyramid/pull/2118 + Deprecations ------------ -- cgit v1.2.3 From 14d86a4bdd6f371966cb9d08e2886bdf59b6e11f Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 23:05:44 -0600 Subject: improve pserve deprecation messages --- pyramid/scripts/pserve.py | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 63f34f6c2..3d55fc4f3 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -69,6 +69,7 @@ class PServeCommand(object): If start/stop/restart is given, then --daemon is implied, and it will start (normal operation), stop (--stop-daemon), or do both. + Note: Daemonization features are deprecated. You can also include variable assignments like 'http_port=8080' and then use %(http_port)s in your config files. @@ -100,13 +101,13 @@ class PServeCommand(object): '--daemon', dest="daemon", action="store_true", - help="Run in daemon (background) mode") + help="Run in daemon (background) mode [DEPRECATED]") parser.add_option( '--pid-file', dest='pid_file', metavar='FILENAME', help=("Save PID to file (default to pyramid.pid if running in " - "daemon mode)")) + "daemon mode) [DEPRECATED]")) parser.add_option( '--log-file', dest='log_file', @@ -127,7 +128,7 @@ class PServeCommand(object): '--monitor-restart', dest='monitor_restart', action='store_true', - help="Auto-restart server if it dies") + help="Auto-restart server if it dies [DEPRECATED]") parser.add_option( '-b', '--browser', dest='browser', @@ -137,7 +138,8 @@ class PServeCommand(object): '--status', action='store_true', dest='show_status', - help="Show the status of the (presumably daemonized) server") + help=("Show the status of the (presumably daemonized) server " + "[DEPRECATED]")) parser.add_option( '-v', '--verbose', default=default_verbosity, @@ -169,7 +171,7 @@ class PServeCommand(object): dest='stop_daemon', action='store_true', help=('Stop a daemonized server (given a PID file, or default ' - 'pyramid.pid file)')) + 'pyramid.pid file) [DEPRECATED]')) _scheme_re = re.compile(r'^[a-z][a-z]+:', re.I) @@ -302,6 +304,17 @@ class PServeCommand(object): raise ValueError(msg) writeable_pid_file.close() + # warn before forking + if ( + self.options.monitor_restart and + not os.environ.get(self._monitor_environ_key) + ): + self.out('''\ +--monitor-restart has been deprecated in Pyramid 1.6. It will be removed +in a future release per Pyramid's deprecation policy. Please consider using +a real process manager for your processes like Systemd, Circus, or Supervisor. +''') + if ( getattr(self.options, 'daemon', False) and not os.environ.get(self._monitor_environ_key) -- cgit v1.2.3 From 54251241c8f52dbe88d6ee21df9f2018fbd7c0d9 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 21:41:07 -0600 Subject: abort when using reload with daemon --- pyramid/scripts/pserve.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 63f34f6c2..42b45640c 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -225,6 +225,10 @@ class PServeCommand(object): cmd = None if self.options.reload: + if self.options.daemon or cmd in ('start', 'stop', 'restart'): + self.out( + 'Error: Cannot use reloading while running as a dameon.') + return 2 if os.environ.get(self._reloader_environ_key): if self.options.verbose > 1: self.out('Running reloading file monitor') -- cgit v1.2.3 From 03465e75ba938a11e9f9c85b7829f7dea9713642 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 23:13:00 -0600 Subject: add changelog for #2119 --- CHANGES.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 77b8e9298..a17c7e1e6 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -235,6 +235,10 @@ Bug Fixes - Fixed usage of ``pserve --monitor-restart --daemon`` which would fail in horrible ways. See https://github.com/Pylons/pyramid/pull/2118 +- Explicitly prevent ``pserve --reload --daemon`` from being used. It's never + been supported but would work and fail in weird ways. + See https://github.com/Pylons/pyramid/pull/2119 + Deprecations ------------ -- cgit v1.2.3 From 203cf3accd0bec0cc08eab8e736f26cd0e711d8b Mon Sep 17 00:00:00 2001 From: Bert JW Regeer Date: Thu, 12 Nov 2015 22:15:51 -0700 Subject: Add AuthTkt test to test ticket timeouts Previous tests did not actually test the code, other than by chance, which meant future changes could regress, which is bad. --- pyramid/tests/test_authentication.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/pyramid/tests/test_authentication.py b/pyramid/tests/test_authentication.py index 4a6525af2..0a22e5965 100644 --- a/pyramid/tests/test_authentication.py +++ b/pyramid/tests/test_authentication.py @@ -769,6 +769,17 @@ class TestAuthTktCookieHelper(unittest.TestCase): helper = self._makeOne('secret', timeout='1') self.assertEqual(helper.timeout, 1) + def test_identify_cookie_timeout_aged(self): + import time + helper = self._makeOne('secret', timeout=10) + now = time.time() + helper.auth_tkt.timestamp = now - 1 + helper.now = now + 10 + helper.auth_tkt.tokens = (text_('a'), ) + request = self._makeRequest('bogus') + result = helper.identify(request) + self.assertFalse(result) + def test_identify_cookie_reissue(self): import time helper = self._makeOne('secret', timeout=10, reissue_time=0) -- cgit v1.2.3 From baff99d69e80c58a76e0b0e8cf444ee5e573208b Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 23:22:00 -0600 Subject: update changelog for #2120 --- CHANGES.txt | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index 77b8e9298..79d610d4a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -238,9 +238,10 @@ Bug Fixes Deprecations ------------ -- The ``pserve`` command's daemonization features have been deprecated. This - includes the ``[start,stop,restart,status]`` subcommands as well as the - ``--daemon``, ``--stop-server``, ``--pid-file``, and ``--status`` flags. +- The ``pserve`` command's daemonization features have been deprecated as well + as ``--monitor-restart``. This includes the ``[start,stop,restart,status]`` + subcommands as well as the ``--daemon``, ``--stop-daemon``, ``--pid-file``, + and ``--status`` flags. Please use a real process manager in the future instead of relying on the ``pserve`` to daemonize itself. Many options exist including your Operating @@ -248,6 +249,7 @@ Deprecations solutions like Circus and Supervisor. See https://github.com/Pylons/pyramid/pull/1641 + and https://github.com/Pylons/pyramid/pull/2120 - Renamed the ``principal`` argument to ``pyramid.security.remember()`` to ``userid`` in order to clarify its intended purpose. -- cgit v1.2.3 From 41587658312a7e53c2be742c320455fa5763bfb0 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 23:42:58 -0600 Subject: fix bug in docstring --- pyramid/static.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyramid/static.py b/pyramid/static.py index 59f440c82..35d5e1047 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -222,7 +222,7 @@ class ManifestCacheBuster(object): Specifically, it is a JSON-serialized dictionary where the keys are the source asset paths used in calls to - :meth:`~pyramid.request.Request.static_url. For example:: + :meth:`~pyramid.request.Request.static_url`. For example:: .. code-block:: python -- cgit v1.2.3 From 93e886be234fd187c4ddc5e376fd6c51060500a7 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Fri, 13 Nov 2015 12:51:00 -0600 Subject: print out every file that has a syntax error --- pyramid/scripts/pserve.py | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 4a6e917a7..1e580d897 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -18,6 +18,7 @@ import py_compile import re import subprocess import sys +import tempfile import textwrap import threading import time @@ -847,8 +848,16 @@ class Monitor(object): # pragma: no cover self.syntax_error_files = set() self.pending_reload = False self.file_callbacks = list(self.global_file_callbacks) + temp_pyc_fp = tempfile.NamedTemporaryFile(delete=False) + self.temp_pyc = temp_pyc_fp.name + temp_pyc_fp.close() def _exit(self): + try: + os.unlink(self.temp_pyc) + except IOError: + # not worried if the tempfile can't be removed + pass # use os._exit() here and not sys.exit() since within a # thread sys.exit() just closes the given thread and # won't kill the process; note os._exit does not call @@ -879,6 +888,7 @@ class Monitor(object): # pragma: no cover continue if filename is not None: filenames.append(filename) + new_changes = False for filename in filenames: try: stat = os.stat(filename) @@ -896,7 +906,7 @@ class Monitor(object): # pragma: no cover old_mtime = self.module_mtimes.get(filename) self.module_mtimes[filename] = mtime if old_mtime is not None and old_mtime < mtime: - self.pending_reload = True + new_changes = True if pyc: filename = filename[:-1] is_valid = True @@ -904,6 +914,11 @@ class Monitor(object): # pragma: no cover is_valid = self.check_syntax(filename) if is_valid: print("%s changed ..." % filename) + if new_changes: + self.pending_reload = True + if self.syntax_error_files: + for filename in sorted(self.syntax_error_files): + print("%s has a SyntaxError; NOT reloading." % filename) if self.pending_reload and not self.syntax_error_files: self.pending_reload = False return False @@ -913,9 +928,9 @@ class Monitor(object): # pragma: no cover # check if a file has syntax errors. # If so, track it until it's fixed. try: - py_compile.compile(filename, doraise=True) - except py_compile.PyCompileError: - print("%s has a SyntaxError; NOT reloading." % filename) + py_compile.compile(filename, cfile=self.temp_pyc, doraise=True) + except py_compile.PyCompileError as ex: + print(ex.msg) self.syntax_error_files.add(filename) return False else: -- cgit v1.2.3 From 9a2761e8739781dde57bbf607926a0c227d9b40f Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Fri, 13 Nov 2015 13:34:38 -0600 Subject: add changelog for #2044 --- CHANGES.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 739eb870d..9ff26420b 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -155,6 +155,9 @@ Features docstrings instead of the default ``str(obj)`` when possible. See https://github.com/Pylons/pyramid/pull/1929 +- ``pserve --reload`` will no longer crash on syntax errors!!! + See https://github.com/Pylons/pyramid/pull/2044 + Bug Fixes --------- -- cgit v1.2.3 From 031b5b69f4ae3e2187e3b9f1f7e582d71fac45b1 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Fri, 13 Nov 2015 15:09:43 -0600 Subject: daemon option may not be set on windows --- pyramid/scripts/pserve.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index b7d4359df..efad0cc68 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -229,7 +229,10 @@ class PServeCommand(object): cmd = None if self.options.reload: - if self.options.daemon or cmd in ('start', 'stop', 'restart'): + if ( + getattr(self.options, 'daemon', False) or + cmd in ('start', 'stop', 'restart') + ): self.out( 'Error: Cannot use reloading while running as a dameon.') return 2 -- cgit v1.2.3 From 63163f7bd71fba313bdab162201cb00a8dcc7c9b Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 13 Nov 2015 13:31:40 -0800 Subject: minor grammar, .rst syntax, rewrap 79 cols --- docs/narr/upgrading.rst | 163 +++++++++++++++++++++++------------------------- 1 file changed, 79 insertions(+), 84 deletions(-) diff --git a/docs/narr/upgrading.rst b/docs/narr/upgrading.rst index eb3194a65..db9b5e090 100644 --- a/docs/narr/upgrading.rst +++ b/docs/narr/upgrading.rst @@ -12,26 +12,26 @@ applications keep working when you upgrade the Pyramid version you're using. .. sidebar:: About Release Numbering Conventionally, application version numbering in Python is described as - ``major.minor.micro``. If your Pyramid version is "1.2.3", it means - you're running a version of Pyramid with the major version "1", the minor - version "2" and the micro version "3". A "major" release is one that - increments the first-dot number; 2.X.X might follow 1.X.X. A "minor" - release is one that increments the second-dot number; 1.3.X might follow - 1.2.X. A "micro" release is one that increments the third-dot number; - 1.2.3 might follow 1.2.2. In general, micro releases are "bugfix-only", - and contain no new features, minor releases contain new features but are - largely backwards compatible with older versions, and a major release - indicates a large set of backwards incompatibilities. + ``major.minor.micro``. If your Pyramid version is "1.2.3", it means you're + running a version of Pyramid with the major version "1", the minor version + "2" and the micro version "3". A "major" release is one that increments the + first-dot number; 2.X.X might follow 1.X.X. A "minor" release is one that + increments the second-dot number; 1.3.X might follow 1.2.X. A "micro" + release is one that increments the third-dot number; 1.2.3 might follow + 1.2.2. In general, micro releases are "bugfix-only", and contain no new + features, minor releases contain new features but are largely backwards + compatible with older versions, and a major release indicates a large set of + backwards incompatibilities. The Pyramid core team is conservative when it comes to removing features. We -don't remove features unnecessarily, but we're human, and we make mistakes -which cause some features to be evolutionary dead ends. Though we are -willing to support dead-end features for some amount of time, some eventually -have to be removed when the cost of supporting them outweighs the benefit of -keeping them around, because each feature in Pyramid represents a certain -documentation and maintenance burden. - -Deprecation and Removal Policy +don't remove features unnecessarily, but we're human and we make mistakes which +cause some features to be evolutionary dead ends. Though we are willing to +support dead-end features for some amount of time, some eventually have to be +removed when the cost of supporting them outweighs the benefit of keeping them +around, because each feature in Pyramid represents a certain documentation and +maintenance burden. + +Deprecation and removal policy ------------------------------ When a feature is scheduled for removal from Pyramid or any of its official @@ -51,50 +51,49 @@ When a deprecated feature is eventually removed: - A note is added to the :ref:`changelog` about the removal. -Features are never removed in *micro* releases. They are only removed in -minor and major releases. Deprecated features are kept around for at least -*three* minor releases from the time the feature became deprecated. -Therefore, if a feature is added in Pyramid 1.0, but it's deprecated in -Pyramid 1.1, it will be kept around through all 1.1.X releases, all 1.2.X -releases and all 1.3.X releases. It will finally be removed in the first -1.4.X release. - -Sometimes features are "docs-deprecated" instead of formally deprecated. -This means that the feature will be kept around indefinitely, but it will be -removed from the documentation or a note will be added to the documentation -telling folks to use some other newer feature. This happens when the cost of -keeping an old feature around is very minimal and the support and -documentation burden is very low. For example, we might rename a function -that is an API without changing the arguments it accepts. In this case, -we'll often rename the function, and change the docs to point at the new -function name, but leave around a backwards compatibility alias to the old -function name so older code doesn't break. +Features are never removed in *micro* releases. They are only removed in minor +and major releases. Deprecated features are kept around for at least *three* +minor releases from the time the feature became deprecated. Therefore, if a +feature is added in Pyramid 1.0, but it's deprecated in Pyramid 1.1, it will be +kept around through all 1.1.X releases, all 1.2.X releases and all 1.3.X +releases. It will finally be removed in the first 1.4.X release. + +Sometimes features are "docs-deprecated" instead of formally deprecated. This +means that the feature will be kept around indefinitely, but it will be removed +from the documentation or a note will be added to the documentation telling +folks to use some other newer feature. This happens when the cost of keeping +an old feature around is very minimal and the support and documentation burden +is very low. For example, we might rename a function that is an API without +changing the arguments it accepts. In this case, we'll often rename the +function, and change the docs to point at the new function name, but leave +around a backwards compatibility alias to the old function name so older code +doesn't break. "Docs deprecated" features tend to work "forever", meaning that they won't be removed, and they'll never generate a deprecation warning. However, such changes are noted in the :ref:`changelog`, so it's possible to know that you -should change older spellings to newer ones to ensure that people reading -your code can find the APIs you're using in the Pyramid docs. +should change older spellings to newer ones to ensure that people reading your +code can find the APIs you're using in the Pyramid docs. -Consulting the Change History +Consulting the change history ----------------------------- -Your first line of defense against application failures caused by upgrading -to a newer Pyramid release is always to read the :ref:`changelog`. to find -the deprecations and removals for each release between the release you're -currently running and the one you wish to upgrade to. The change history -notes every deprecation within a ``Deprecation`` section and every removal -within a ``Backwards Incompatibilies`` section for each release. +Your first line of defense against application failures caused by upgrading to +a newer Pyramid release is always to read the :ref:`changelog` to find the +deprecations and removals for each release between the release you're currently +running and the one to which you wish to upgrade. The change history notes +every deprecation within a ``Deprecation`` section and every removal within a +``Backwards Incompatibilies`` section for each release. -The change history often contains instructions for changing your code to -avoid deprecation warnings and how to change docs-deprecated spellings to -newer ones. You can follow along with each deprecation explanation in the -change history, simply doing a grep or other code search to your application, -using the change log examples to remediate each potential problem. +The change history often contains instructions for changing your code to avoid +deprecation warnings and how to change docs-deprecated spellings to newer ones. +You can follow along with each deprecation explanation in the change history, +simply doing a grep or other code search to your application, using the change +log examples to remediate each potential problem. .. _testing_under_new_release: -Testing Your Application Under a New Pyramid Release +Testing your application under a new Pyramid release ---------------------------------------------------- Once you've upgraded your application to a new Pyramid release and you've @@ -106,25 +105,24 @@ you can see DeprecationWarnings printed to the console when the tests run. $ python -Wd setup.py test -q -The ``-Wd`` argument is an argument that tells Python to print deprecation -warnings to the console. Note that the ``-Wd`` flag is only required for -Python 2.7 and better: Python versions 2.6 and older print deprecation -warnings to the console by default. See `the Python -W flag documentation -`_ for more -information. +The ``-Wd`` argument tells Python to print deprecation warnings to the console. +Note that the ``-Wd`` flag is only required for Python 2.7 and better: Python +versions 2.6 and older print deprecation warnings to the console by default. +See `the Python -W flag documentation +`_ for more information. As your tests run, deprecation warnings will be printed to the console -explaining the deprecation and providing instructions about how to prevent -the deprecation warning from being issued. For example: +explaining the deprecation and providing instructions about how to prevent the +deprecation warning from being issued. For example: -.. code-block:: text +.. code-block:: bash $ python -Wd setup.py test -q # .. elided ... running build_ext - /home/chrism/projects/pyramid/env27/myproj/myproj/views.py:3: - DeprecationWarning: static: The "pyramid.view.static" class is deprecated - as of Pyramid 1.1; use the "pyramid.static.static_view" class instead with + /home/chrism/projects/pyramid/env27/myproj/myproj/views.py:3: + DeprecationWarning: static: The "pyramid.view.static" class is deprecated + as of Pyramid 1.1; use the "pyramid.static.static_view" class instead with the "use_subpath" argument set to True. from pyramid.view import static . @@ -144,8 +142,8 @@ pyramid.view import static``) that is causing the problem: from pyramid.view import static myview = static('static', 'static') -The deprecation warning tells me how to fix it, so I can change the code to -do things the newer way: +The deprecation warning tells me how to fix it, so I can change the code to do +things the newer way: .. code-block:: python :linenos: @@ -155,10 +153,10 @@ do things the newer way: from pyramid.static import static_view myview = static_view('static', 'static', use_subpath=True) -When I run the tests again, the deprecation warning is no longer printed to -my console: +When I run the tests again, the deprecation warning is no longer printed to my +console: -.. code-block:: text +.. code-block:: bash $ python -Wd setup.py test -q # .. elided ... @@ -170,7 +168,7 @@ my console: OK -My Application Doesn't Have Any Tests or Has Few Tests +My application doesn't have any tests or has few tests ------------------------------------------------------ If your application has no tests, or has only moderate test coverage, running @@ -178,8 +176,8 @@ tests won't tell you very much, because the Pyramid codepaths that generate deprecation warnings won't be executed. In this circumstance, you can start your application interactively under a -server run with the ``PYTHONWARNINGS`` environment variable set to -``default``. On UNIX, you can do that via: +server run with the ``PYTHONWARNINGS`` environment variable set to ``default``. +On UNIX, you can do that via: .. code-block:: bash @@ -194,16 +192,15 @@ On Windows, you need to issue two commands: At this point, it's ensured that deprecation warnings will be printed to the console whenever a codepath is hit that generates one. You can then click -around in your application interactively to try to generate them, and -remediate as explained in :ref:`testing_under_new_release`. +around in your application interactively to try to generate them, and remediate +as explained in :ref:`testing_under_new_release`. See `the PYTHONWARNINGS environment variable documentation `_ or `the Python -W flag documentation -`_ for more -information. +`_ for more information. -Upgrading to the Very Latest Pyramid Release +Upgrading to the very latest Pyramid release -------------------------------------------- When you upgrade your application to the most recent Pyramid release, @@ -220,15 +217,13 @@ advisable to do this: :ref:`testing_under_new_release`. Note any deprecation warnings and remediate. -- Upgrade to the most recent 1.3 release, 1.3.3. Run your application's - tests, note any deprecation warnings and remediate. +- Upgrade to the most recent 1.3 release, 1.3.3. Run your application's tests, + note any deprecation warnings, and remediate. - Upgrade to 1.4.4. Run your application's tests, note any deprecation - warnings and remediate. + warnings, and remediate. If you skip testing your application under each minor release (for example if -you upgrade directly from 1.2.1 to 1.4.4), you might miss a deprecation -warning and waste more time trying to figure out an error caused by a feature -removal than it would take to upgrade stepwise through each minor release. - - +you upgrade directly from 1.2.1 to 1.4.4), you might miss a deprecation warning +and waste more time trying to figure out an error caused by a feature removal +than it would take to upgrade stepwise through each minor release. -- cgit v1.2.3 From bdc3f28699c5fc6b127a1aa502b8372b149429f2 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Sun, 15 Nov 2015 05:46:47 -0800 Subject: minor grammar, .rst syntax fixes, rewrap 79 cols --- docs/narr/threadlocals.rst | 226 +++++++++++++++++++++------------------------ 1 file changed, 106 insertions(+), 120 deletions(-) diff --git a/docs/narr/threadlocals.rst b/docs/narr/threadlocals.rst index afe56de3e..7437a3a76 100644 --- a/docs/narr/threadlocals.rst +++ b/docs/narr/threadlocals.rst @@ -8,26 +8,24 @@ Thread Locals ============= -A :term:`thread local` variable is a variable that appears to be a -"global" variable to an application which uses it. However, unlike a -true global variable, one thread or process serving the application -may receive a different value than another thread or process when that -variable is "thread local". +A :term:`thread local` variable is a variable that appears to be a "global" +variable to an application which uses it. However, unlike a true global +variable, one thread or process serving the application may receive a different +value than another thread or process when that variable is "thread local". -When a request is processed, :app:`Pyramid` makes two :term:`thread -local` variables available to the application: a "registry" and a -"request". +When a request is processed, :app:`Pyramid` makes two :term:`thread local` +variables available to the application: a "registry" and a "request". Why and How :app:`Pyramid` Uses Thread Local Variables ---------------------------------------------------------- +------------------------------------------------------ -How are thread locals beneficial to :app:`Pyramid` and application -developers who use :app:`Pyramid`? Well, usually they're decidedly -**not**. Using a global or a thread local variable in any application -usually makes it a lot harder to understand for a casual reader. Use -of a thread local or a global is usually just a way to avoid passing -some value around between functions, which is itself usually a very -bad idea, at least if code readability counts as an important concern. +How are thread locals beneficial to :app:`Pyramid` and application developers +who use :app:`Pyramid`? Well, usually they're decidedly **not**. Using a +global or a thread local variable in any application usually makes it a lot +harder to understand for a casual reader. Use of a thread local or a global is +usually just a way to avoid passing some value around between functions, which +is itself usually a very bad idea, at least if code readability counts as an +important concern. For historical reasons, however, thread local variables are indeed consulted by various :app:`Pyramid` API functions. For example, the implementation of the @@ -40,119 +38,107 @@ application registry, from which it looks up the authentication policy; it then uses the authentication policy to retrieve the authenticated user id. This is how :app:`Pyramid` allows arbitrary authentication policies to be "plugged in". -When they need to do so, :app:`Pyramid` internals use two API -functions to retrieve the :term:`request` and :term:`application -registry`: :func:`~pyramid.threadlocal.get_current_request` and -:func:`~pyramid.threadlocal.get_current_registry`. The former -returns the "current" request; the latter returns the "current" -registry. Both ``get_current_*`` functions retrieve an object from a -thread-local data structure. These API functions are documented in -:ref:`threadlocal_module`. - -These values are thread locals rather than true globals because one -Python process may be handling multiple simultaneous requests or even -multiple :app:`Pyramid` applications. If they were true globals, -:app:`Pyramid` could not handle multiple simultaneous requests or -allow more than one :app:`Pyramid` application instance to exist in -a single Python process. - -Because one :app:`Pyramid` application is permitted to call -*another* :app:`Pyramid` application from its own :term:`view` code -(perhaps as a :term:`WSGI` app with help from the -:func:`pyramid.wsgi.wsgiapp2` decorator), these variables are -managed in a *stack* during normal system operations. The stack -instance itself is a :class:`threading.local`. +When they need to do so, :app:`Pyramid` internals use two API functions to +retrieve the :term:`request` and :term:`application registry`: +:func:`~pyramid.threadlocal.get_current_request` and +:func:`~pyramid.threadlocal.get_current_registry`. The former returns the +"current" request; the latter returns the "current" registry. Both +``get_current_*`` functions retrieve an object from a thread-local data +structure. These API functions are documented in :ref:`threadlocal_module`. + +These values are thread locals rather than true globals because one Python +process may be handling multiple simultaneous requests or even multiple +:app:`Pyramid` applications. If they were true globals, :app:`Pyramid` could +not handle multiple simultaneous requests or allow more than one :app:`Pyramid` +application instance to exist in a single Python process. + +Because one :app:`Pyramid` application is permitted to call *another* +:app:`Pyramid` application from its own :term:`view` code (perhaps as a +:term:`WSGI` app with help from the :func:`pyramid.wsgi.wsgiapp2` decorator), +these variables are managed in a *stack* during normal system operations. The +stack instance itself is a :class:`threading.local`. During normal operations, the thread locals stack is managed by a -:term:`Router` object. At the beginning of a request, the Router -pushes the application's registry and the request on to the stack. At -the end of a request, the stack is popped. The topmost request and -registry on the stack are considered "current". Therefore, when the -system is operating normally, the very definition of "current" is -defined entirely by the behavior of a pyramid :term:`Router`. +:term:`Router` object. At the beginning of a request, the Router pushes the +application's registry and the request on to the stack. At the end of a +request, the stack is popped. The topmost request and registry on the stack +are considered "current". Therefore, when the system is operating normally, +the very definition of "current" is defined entirely by the behavior of a +pyramid :term:`Router`. However, during unit testing, no Router code is ever invoked, and the -definition of "current" is defined by the boundary between calls to -the :meth:`pyramid.config.Configurator.begin` and -:meth:`pyramid.config.Configurator.end` methods (or between -calls to the :func:`pyramid.testing.setUp` and -:func:`pyramid.testing.tearDown` functions). These functions push -and pop the threadlocal stack when the system is under test. See -:ref:`test_setup_and_teardown` for the definitions of these functions. - -Scripts which use :app:`Pyramid` machinery but never actually start -a WSGI server or receive requests via HTTP such as scripts which use -the :mod:`pyramid.scripting` API will never cause any Router code -to be executed. However, the :mod:`pyramid.scripting` APIs also -push some values on to the thread locals stack as a matter of course. -Such scripts should expect the -:func:`~pyramid.threadlocal.get_current_request` function to always -return ``None``, and should expect the -:func:`~pyramid.threadlocal.get_current_registry` function to return -exactly the same :term:`application registry` for every request. +definition of "current" is defined by the boundary between calls to the +:meth:`pyramid.config.Configurator.begin` and +:meth:`pyramid.config.Configurator.end` methods (or between calls to the +:func:`pyramid.testing.setUp` and :func:`pyramid.testing.tearDown` functions). +These functions push and pop the threadlocal stack when the system is under +test. See :ref:`test_setup_and_teardown` for the definitions of these +functions. + +Scripts which use :app:`Pyramid` machinery but never actually start a WSGI +server or receive requests via HTTP, such as scripts which use the +:mod:`pyramid.scripting` API, will never cause any Router code to be executed. +However, the :mod:`pyramid.scripting` APIs also push some values on to the +thread locals stack as a matter of course. Such scripts should expect the +:func:`~pyramid.threadlocal.get_current_request` function to always return +``None``, and should expect the +:func:`~pyramid.threadlocal.get_current_registry` function to return exactly +the same :term:`application registry` for every request. Why You Shouldn't Abuse Thread Locals ------------------------------------- You probably should almost never use the :func:`~pyramid.threadlocal.get_current_request` or -:func:`~pyramid.threadlocal.get_current_registry` functions, except -perhaps in tests. In particular, it's almost always a mistake to use -``get_current_request`` or ``get_current_registry`` in application -code because its usage makes it possible to write code that can be -neither easily tested nor scripted. Inappropriate usage is defined as -follows: +:func:`~pyramid.threadlocal.get_current_registry` functions, except perhaps in +tests. In particular, it's almost always a mistake to use +``get_current_request`` or ``get_current_registry`` in application code because +its usage makes it possible to write code that can be neither easily tested nor +scripted. Inappropriate usage is defined as follows: - ``get_current_request`` should never be called within the body of a - :term:`view callable`, or within code called by a view callable. - View callables already have access to the request (it's passed in to - each as ``request``). - -- ``get_current_request`` should never be called in :term:`resource` code. - If a resource needs access to the request, it should be passed the request - by a :term:`view callable`. - -- ``get_current_request`` function should never be called because it's - "easier" or "more elegant" to think about calling it than to pass a - request through a series of function calls when creating some API - design. Your application should instead almost certainly pass data - derived from the request around rather than relying on being able to - call this function to obtain the request in places that actually - have no business knowing about it. Parameters are *meant* to be - passed around as function arguments, this is why they exist. Don't - try to "save typing" or create "nicer APIs" by using this function - in the place where a request is required; this will only lead to - sadness later. - -- Neither ``get_current_request`` nor ``get_current_registry`` should - ever be called within application-specific forks of third-party - library code. The library you've forked almost certainly has - nothing to do with :app:`Pyramid`, and making it dependent on - :app:`Pyramid` (rather than making your :app:`pyramid` - application depend upon it) means you're forming a dependency in the - wrong direction. - -Use of the :func:`~pyramid.threadlocal.get_current_request` function -in application code *is* still useful in very limited circumstances. -As a rule of thumb, usage of ``get_current_request`` is useful -**within code which is meant to eventually be removed**. For -instance, you may find yourself wanting to deprecate some API that -expects to be passed a request object in favor of one that does not -expect to be passed a request object. But you need to keep -implementations of the old API working for some period of time while -you deprecate the older API. So you write a "facade" implementation -of the new API which calls into the code which implements the older -API. Since the new API does not require the request, your facade -implementation doesn't have local access to the request when it needs -to pass it into the older API implementation. After some period of -time, the older implementation code is disused and the hack that uses -``get_current_request`` is removed. This would be an appropriate -place to use the ``get_current_request``. - -Use of the :func:`~pyramid.threadlocal.get_current_registry` -function should be limited to testing scenarios. The registry made -current by use of the -:meth:`pyramid.config.Configurator.begin` method during a -test (or via :func:`pyramid.testing.setUp`) when you do not pass -one in is available to you via this API. - + :term:`view callable`, or within code called by a view callable. View + callables already have access to the request (it's passed in to each as + ``request``). + +- ``get_current_request`` should never be called in :term:`resource` code. If a + resource needs access to the request, it should be passed the request by a + :term:`view callable`. + +- ``get_current_request`` function should never be called because it's "easier" + or "more elegant" to think about calling it than to pass a request through a + series of function calls when creating some API design. Your application + should instead, almost certainly, pass around data derived from the request + rather than relying on being able to call this function to obtain the request + in places that actually have no business knowing about it. Parameters are + *meant* to be passed around as function arguments; this is why they exist. + Don't try to "save typing" or create "nicer APIs" by using this function in + the place where a request is required; this will only lead to sadness later. + +- Neither ``get_current_request`` nor ``get_current_registry`` should ever be + called within application-specific forks of third-party library code. The + library you've forked almost certainly has nothing to do with :app:`Pyramid`, + and making it dependent on :app:`Pyramid` (rather than making your + :app:`pyramid` application depend upon it) means you're forming a dependency + in the wrong direction. + +Use of the :func:`~pyramid.threadlocal.get_current_request` function in +application code *is* still useful in very limited circumstances. As a rule of +thumb, usage of ``get_current_request`` is useful **within code which is meant +to eventually be removed**. For instance, you may find yourself wanting to +deprecate some API that expects to be passed a request object in favor of one +that does not expect to be passed a request object. But you need to keep +implementations of the old API working for some period of time while you +deprecate the older API. So you write a "facade" implementation of the new API +which calls into the code which implements the older API. Since the new API +does not require the request, your facade implementation doesn't have local +access to the request when it needs to pass it into the older API +implementation. After some period of time, the older implementation code is +disused and the hack that uses ``get_current_request`` is removed. This would +be an appropriate place to use the ``get_current_request``. + +Use of the :func:`~pyramid.threadlocal.get_current_registry` function should be +limited to testing scenarios. The registry made current by use of the +:meth:`pyramid.config.Configurator.begin` method during a test (or via +:func:`pyramid.testing.setUp`) when you do not pass one in is available to you +via this API. -- cgit v1.2.3 From b9f8dcfdb745c81a437549922f04d03ae7f45614 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 15 Nov 2015 19:53:54 -0600 Subject: add .exe to the script being invoked if missing on windows fixes #2126 --- pyramid/scripts/pserve.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index efad0cc68..95752a3be 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -428,6 +428,19 @@ a real process manager for your processes like Systemd, Circus, or Supervisor. arg = win32api.GetShortPathName(arg) return arg + def find_script_path(self, name): # pragma: no cover + """ + Return the path to the script being invoked by the python interpreter. + + There's an issue on Windows when running the executable from + a console_script causing the script name (sys.argv[0]) to + not end with .exe or .py and thus cannot be run via popen. + """ + if sys.platform == 'win32': + if not name.endswith('.exe') and not name.endswith('.py'): + name += '.exe' + return name + def daemonize(self): # pragma: no cover pid = live_pidfile(self.options.pid_file) if pid: @@ -573,7 +586,10 @@ a real process manager for your processes like Systemd, Circus, or Supervisor. else: self.out('Starting subprocess with monitor parent') while 1: - args = [self.quote_first_command_arg(sys.executable)] + sys.argv + args = [ + self.quote_first_command_arg(sys.executable), + self.find_script_path(sys.argv[0]), + ] + sys.argv[1:] new_environ = os.environ.copy() if reloader: new_environ[self._reloader_environ_key] = 'true' -- cgit v1.2.3 From a8c0d7161f12a24bc439495b5f1af78da3dfcf17 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 15 Nov 2015 20:01:03 -0600 Subject: update changelog --- CHANGES.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 9ff26420b..e9dc975a7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -247,6 +247,10 @@ Bug Fixes been supported but would work and fail in weird ways. See https://github.com/Pylons/pyramid/pull/2119 +- Fix an issue on Windows when running ``pserve --reload`` in which the + process failed to fork because it could not find the pserve script to + run. See https://github.com/Pylons/pyramid/pull/2137 + Deprecations ------------ -- cgit v1.2.3 From 1e1111ba774c4cb12b075338e921283047bd3600 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 12 Nov 2015 10:45:47 -0600 Subject: support asset specs in ManifestCacheBuster --- pyramid/static.py | 20 +++++++++++++++----- pyramid/tests/test_static.py | 14 ++++++++++++++ 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/pyramid/static.py b/pyramid/static.py index 35d5e1047..c2c8c89e5 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -19,7 +19,10 @@ from pkg_resources import ( from repoze.lru import lru_cache -from pyramid.asset import resolve_asset_spec +from pyramid.asset import ( + abspath_from_asset_spec, + resolve_asset_spec, +) from pyramid.compat import text_ @@ -211,7 +214,11 @@ class ManifestCacheBuster(object): uses a supplied manifest file to map an asset path to a cache-busted version of the path. - The file is expected to conform to the following simple JSON format: + The ``manifest_spec`` can be an absolute path or a :term:`asset spec` + pointing to a package-relative file. + + The manifest file is expected to conform to the following simple JSON + format: .. code-block:: json @@ -247,8 +254,10 @@ class ManifestCacheBuster(object): exists = staticmethod(exists) # testing getmtime = staticmethod(getmtime) # testing - def __init__(self, manifest_path, reload=False): - self.manifest_path = manifest_path + def __init__(self, manifest_spec, reload=False): + package_name = caller_package().__name__ + self.manifest_path = abspath_from_asset_spec( + manifest_spec, package_name) self.reload = reload self._mtime = None @@ -260,7 +269,8 @@ class ManifestCacheBuster(object): Return a mapping parsed from the ``manifest_path``. Subclasses may override this method to use something other than - ``json.loads``. + ``json.loads`` to load any type of file format and return a conforming + dictionary. """ with open(self.manifest_path, 'rb') as fp: diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index ac30e9e50..4a07c2cb1 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -423,6 +423,20 @@ class TestManifestCacheBuster(unittest.TestCase): fut('foo', ('css', 'main.css'), {}), (['css', 'main-test.css'], {})) + def test_it_with_relspec(self): + fut = self._makeOne('fixtures/manifest.json').pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + + def test_it_with_absspec(self): + fut = self._makeOne('pyramid.tests:fixtures/manifest.json').pregenerate + self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + self.assertEqual( + fut('foo', ('css', 'main.css'), {}), + (['css', 'main-test.css'], {})) + def test_reload(self): manifest_path = os.path.join(here, 'fixtures', 'manifest.json') new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') -- cgit v1.2.3 From 104870609e23edd1dba4d64869a068e883767552 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 15 Nov 2015 21:59:56 -0600 Subject: update docs to use asset specs with ManifestCacheBuster --- docs/narr/assets.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index d36fa49c0..0f819570c 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -500,15 +500,12 @@ The following code would set up a cachebuster: .. code-block:: python :linenos: - from pyramid.path import AssetResolver from pyramid.static import ManifestCacheBuster - resolver = AssetResolver() - manifest = resolver.resolve('myapp:static/manifest.json') config.add_static_view( name='http://mycdn.example.com/', path='mypackage:static', - cachebust=ManifestCacheBuster(manifest.abspath())) + cachebust=ManifestCacheBuster('myapp:static/manifest.json')) A simpler approach is to use the :class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global -- cgit v1.2.3 From 3fe24b103996bc5254a4734e15a5320da84bd76e Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 20 Nov 2015 19:00:44 -0800 Subject: spell out "specification" so that sphinx will not emit a warning for not being able to find "asset spec" --- pyramid/static.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pyramid/static.py b/pyramid/static.py index c2c8c89e5..c7a5c7ba5 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -214,8 +214,8 @@ class ManifestCacheBuster(object): uses a supplied manifest file to map an asset path to a cache-busted version of the path. - The ``manifest_spec`` can be an absolute path or a :term:`asset spec` - pointing to a package-relative file. + The ``manifest_spec`` can be an absolute path or a :term:`asset + specification` pointing to a package-relative file. The manifest file is expected to conform to the following simple JSON format: -- cgit v1.2.3 From a3f0a397300aa290ec213760e11f85f40faa1bd7 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 20 Nov 2015 19:21:44 -0800 Subject: use intersphinx links to webob objects --- docs/designdefense.rst | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/docs/designdefense.rst b/docs/designdefense.rst index 1ed4f65a4..ee6d5a317 100644 --- a/docs/designdefense.rst +++ b/docs/designdefense.rst @@ -907,23 +907,22 @@ creating a more Zope3-like environment without much effort. .. _http_exception_hierarchy: -Pyramid Uses its Own HTTP Exception Class Hierarchy Rather Than ``webob.exc`` ------------------------------------------------------------------------------ +Pyramid uses its own HTTP exception class hierarchy rather than :mod:`webob.exc` +-------------------------------------------------------------------------------- .. versionadded:: 1.1 The HTTP exception classes defined in :mod:`pyramid.httpexceptions` are very -much like the ones defined in ``webob.exc`` -(e.g. :class:`~pyramid.httpexceptions.HTTPNotFound`, -:class:`~pyramid.httpexceptions.HTTPForbidden`, etc). They have the same -names and largely the same behavior and all have a very similar -implementation, but not the same identity. Here's why they have a separate -identity: +much like the ones defined in :mod:`webob.exc`, (e.g., +:class:`~pyramid.httpexceptions.HTTPNotFound` or +:class:`~pyramid.httpexceptions.HTTPForbidden`). They have the same names and +largely the same behavior, and all have a very similar implementation, but not +the same identity. Here's why they have a separate identity: - Making them separate allows the HTTP exception classes to subclass :class:`pyramid.response.Response`. This speeds up response generation - slightly due to the way the Pyramid router works. The same speedup could - be gained by monkeypatching ``webob.response.Response`` but it's usually + slightly due to the way the Pyramid router works. The same speedup could be + gained by monkeypatching :class:`webob.response.Response`, but it's usually the case that monkeypatching turns out to be evil and wrong. - Making them separate allows them to provide alternate ``__call__`` logic @@ -933,7 +932,7 @@ identity: value of ``RequestClass`` (:class:`pyramid.request.Request`). - Making them separate allows us freedom from having to think about backwards - compatibility code present in ``webob.exc`` having to do with Python 2.4, + compatibility code present in :mod:`webob.exc` having to do with Python 2.4, which we no longer support in Pyramid 1.1+. - We change the behavior of two classes @@ -944,9 +943,9 @@ identity: - Making them separate allows us to influence the docstrings of the exception classes to provide Pyramid-specific documentation. -- Making them separate allows us to silence a stupid deprecation warning - under Python 2.6 when the response objects are used as exceptions (related - to ``self.message``). +- Making them separate allows us to silence a stupid deprecation warning under + Python 2.6 when the response objects are used as exceptions (related to + ``self.message``). .. _simpler_traversal_model: -- cgit v1.2.3 From 84a168ef5fd4bdd487226f43b8c0e16237e82e18 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 20 Nov 2015 19:38:26 -0800 Subject: add optional step to update the Pyramid Dash docset --- RELEASING.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/RELEASING.txt b/RELEASING.txt index 87ff62c53..0ed9d5692 100644 --- a/RELEASING.txt +++ b/RELEASING.txt @@ -38,6 +38,9 @@ Releasing Pyramid - Change CHANGES.txt heading to reflect the new version number. +- (Optional) Update the Pyramid Dash docset. + https://github.com/Kapeli/Dash-User-Contributions/tree/master/docsets/Pyramid + - Make sure PyPI long description renders (requires ``collective.dist`` installed into your Python):: -- cgit v1.2.3 From ee9c620963553a3a959cdfc517f1e0818a21e9c0 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 23 Nov 2015 12:59:55 -0600 Subject: expose the PickleSerializer --- docs/api/session.rst | 1 + pyramid/session.py | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/api/session.rst b/docs/api/session.rst index dde9d20e9..474e2bb32 100644 --- a/docs/api/session.rst +++ b/docs/api/session.rst @@ -17,4 +17,5 @@ .. autofunction:: BaseCookieSessionFactory + .. autoclass:: PickleSerializer diff --git a/pyramid/session.py b/pyramid/session.py index fa85fe69c..51f9de620 100644 --- a/pyramid/session.py +++ b/pyramid/session.py @@ -133,13 +133,25 @@ def check_csrf_token(request, return True class PickleSerializer(object): - """ A Webob cookie serializer that uses the pickle protocol to dump Python - data to bytes.""" + """ A serializer that uses the pickle protocol to dump Python + data to bytes. + + This is the default serializer used by Pyramid. + + ``protocol`` may be specified to control the version of pickle used. + Defaults to :attr:`pickle.HIGHEST_PROTOCOL`. + + """ + def __init__(self, protocol=pickle.HIGHEST_PROTOCOL): + self.protocol = protocol + def loads(self, bstruct): + """Accept bytes and return a Python object.""" return pickle.loads(bstruct) def dumps(self, appstruct): - return pickle.dumps(appstruct, pickle.HIGHEST_PROTOCOL) + """Accept a Python object and return bytes.""" + return pickle.dumps(appstruct, self.protocol) def BaseCookieSessionFactory( serializer, -- cgit v1.2.3 From a708d359ff123084ea64b2e53c3ad32a74711219 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 25 Nov 2015 18:52:54 -0600 Subject: remove py2-docs from tox.ini and reorder so coverage is last --- HACKING.txt | 2 +- builddocs.sh | 2 +- tox.ini | 87 ++++++++++++++++++++++++++++-------------------------------- 3 files changed, 43 insertions(+), 48 deletions(-) diff --git a/HACKING.txt b/HACKING.txt index d0f9a769e..c838fda22 100644 --- a/HACKING.txt +++ b/HACKING.txt @@ -217,7 +217,7 @@ changed to reflect the bug fix, ideally in the same commit that fixes the bug or adds the feature. To build and review docs, use the following steps. 1. In the main Pyramid checkout directory, run ``./builddocs.sh`` (which just - turns around and runs ``tox -e py2-docs,py3-docs``):: + turns around and runs ``tox -e docs``):: $ ./builddocs.sh diff --git a/builddocs.sh b/builddocs.sh index eaf02fc1d..0859fe268 100755 --- a/builddocs.sh +++ b/builddocs.sh @@ -1,3 +1,3 @@ #!/bin/bash -tox -epy2-docs,py3-docs +tox -e docs diff --git a/tox.ini b/tox.ini index 20a9ee5b1..626931faf 100644 --- a/tox.ini +++ b/tox.ini @@ -1,7 +1,7 @@ [tox] envlist = - py26,py27,py32,py33,py34,py35,pypy,pypy3,pep8, - {py2,py3}-docs, + py26,py27,py32,py33,py34,py35,pypy,pypy3, + docs,pep8, {py2,py3}-cover,coverage, [testenv] @@ -23,49 +23,6 @@ commands = pip install pyramid[testing] nosetests --with-xunit --xunit-file=nosetests-{envname}.xml {posargs:} -# we separate coverage into its own testenv because a) "last run wins" wrt -# cobertura jenkins reporting and b) pypy and jython can't handle any -# combination of versions of coverage and nosexcover that i can find. -[testenv:py2-cover] -commands = - pip install pyramid[testing] - coverage run --source=pyramid {envbindir}/nosetests - coverage xml -o coverage-py2.xml -setenv = - COVERAGE_FILE=.coverage.py2 - -[testenv:py3-cover] -commands = - pip install pyramid[testing] - coverage run --source=pyramid {envbindir}/nosetests - coverage xml -o coverage-py3.xml -setenv = - COVERAGE_FILE=.coverage.py3 - -[testenv:coverage] -basepython = python3.4 -commands = - coverage erase - coverage combine - coverage xml - coverage report --show-missing --fail-under=100 -deps = - coverage -setenv = - COVERAGE_FILE=.coverage - -[testenv:py2-docs] -whitelist_externals = make -commands = - pip install pyramid[docs] - make -C docs html epub BUILDDIR={envdir} "SPHINXOPTS=-W -E" - -[testenv:py3-docs] -whitelist_externals = make -commands = - pip install pyramid[docs] - make -C docs html epub BUILDDIR={envdir} "SPHINXOPTS=-W -E" - [testenv:py26-scaffolds] basepython = python2.6 commands = @@ -109,8 +66,46 @@ commands = deps = virtualenv [testenv:pep8] -basepython = python3.4 +basepython = python3.5 commands = flake8 pyramid/ deps = flake8 + +[testenv:docs] +basepython = python3.5 +whitelist_externals = make +commands = + pip install pyramid[docs] + make -C docs html epub BUILDDIR={envdir} "SPHINXOPTS=-W -E" + +# we separate coverage into its own testenv because a) "last run wins" wrt +# cobertura jenkins reporting and b) pypy and jython can't handle any +# combination of versions of coverage and nosexcover that i can find. +[testenv:py2-cover] +commands = + pip install pyramid[testing] + coverage run --source=pyramid {envbindir}/nosetests + coverage xml -o coverage-py2.xml +setenv = + COVERAGE_FILE=.coverage.py2 + +[testenv:py3-cover] +commands = + pip install pyramid[testing] + coverage run --source=pyramid {envbindir}/nosetests + coverage xml -o coverage-py3.xml +setenv = + COVERAGE_FILE=.coverage.py3 + +[testenv:coverage] +basepython = python3.5 +commands = + coverage erase + coverage combine + coverage xml + coverage report --show-missing --fail-under=100 +deps = + coverage +setenv = + COVERAGE_FILE=.coverage -- cgit v1.2.3 From 0030fba497a48e596167ceffb6dd499d67c91765 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 25 Nov 2015 18:54:39 -0600 Subject: add docs to travis builds --- .travis.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.travis.yml b/.travis.yml index 79d9fa09d..2163eb8fd 100644 --- a/.travis.yml +++ b/.travis.yml @@ -22,6 +22,8 @@ matrix: env: TOXENV=pypy3 - python: 3.5 env: TOXENV=py2-cover,py3-cover,coverage + - python: 3.5 + env: TOXENV=docs - python: 3.5 env: TOXENV=pep8 -- cgit v1.2.3 From a398ea813923040eea7cb9bcaced55b982149708 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Sat, 28 Nov 2015 19:46:34 -0800 Subject: revert optional step for Dash docset --- RELEASING.txt | 3 --- 1 file changed, 3 deletions(-) diff --git a/RELEASING.txt b/RELEASING.txt index 0ed9d5692..87ff62c53 100644 --- a/RELEASING.txt +++ b/RELEASING.txt @@ -38,9 +38,6 @@ Releasing Pyramid - Change CHANGES.txt heading to reflect the new version number. -- (Optional) Update the Pyramid Dash docset. - https://github.com/Kapeli/Dash-User-Contributions/tree/master/docsets/Pyramid - - Make sure PyPI long description renders (requires ``collective.dist`` installed into your Python):: -- cgit v1.2.3 From 6e29b425182ccc4abc87fcfb32e20b60b15d4bdf Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 1 Dec 2015 01:59:14 -0600 Subject: initial work on config.add_cache_buster --- pyramid/config/views.py | 108 ++++++++++++++++++-------------- pyramid/interfaces.py | 45 ++++--------- pyramid/static.py | 10 ++- pyramid/tests/test_config/test_views.py | 66 +++++++++---------- 4 files changed, 106 insertions(+), 123 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index e386bc4e1..67a70145c 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1,3 +1,4 @@ +import bisect import inspect import operator import os @@ -1855,18 +1856,7 @@ class ViewsConfiguratorMixin(object): ``Expires`` and ``Cache-Control`` headers for static assets served. Note that this argument has no effect when the ``name`` is a *url prefix*. By default, this argument is ``None``, meaning that no - particular Expires or Cache-Control headers are set in the response, - unless ``cachebust`` is specified. - - The ``cachebust`` keyword argument may be set to cause - :meth:`~pyramid.request.Request.static_url` to use cache busting when - generating URLs. See :ref:`cache_busting` for general information - about cache busting. The value of the ``cachebust`` argument must - be an object which implements - :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` - argument is provided, the default for ``cache_max_age`` is modified - to be ten years. ``cache_max_age`` may still be explicitly provided - to override this default. + particular Expires or Cache-Control headers are set in the response. The ``permission`` keyword argument is used to specify the :term:`permission` required by a user to execute the static view. By @@ -1946,11 +1936,32 @@ class ViewsConfiguratorMixin(object): See :ref:`static_assets_section` for more information. """ spec = self._make_spec(path) + info = self._get_static_info() + info.add(self, name, spec, **kw) + + def add_cache_buster(self, path, cachebust): + """ + The ``cachebust`` keyword argument may be set to cause + :meth:`~pyramid.request.Request.static_url` to use cache busting when + generating URLs. See :ref:`cache_busting` for general information + about cache busting. The value of the ``cachebust`` argument must + be an object which implements + :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` + argument is provided, the default for ``cache_max_age`` is modified + to be ten years. ``cache_max_age`` may still be explicitly provided + to override this default. + + """ + spec = self._make_spec(path) + info = self._get_static_info() + info.add_cache_buster(self, spec, cachebust) + + def _get_static_info(self): info = self.registry.queryUtility(IStaticURLInfo) if info is None: info = StaticURLInfo() self.registry.registerUtility(info, IStaticURLInfo) - info.add(self, name, spec, **kw) + return info def isexception(o): if IInterface.providedBy(o): @@ -1964,26 +1975,18 @@ def isexception(o): @implementer(IStaticURLInfo) class StaticURLInfo(object): - def _get_registrations(self, registry): - try: - reg = registry._static_url_registrations - except AttributeError: - reg = registry._static_url_registrations = [] - return reg + def __init__(self): + self.registrations = [] + self.cache_busters = [] def generate(self, path, request, **kw): - try: - registry = request.registry - except AttributeError: # bw compat (for tests) - registry = get_current_registry() - registrations = self._get_registrations(registry) - for (url, spec, route_name, cachebust) in registrations: + for (url, spec, route_name) in self.registrations: if path.startswith(spec): subpath = path[len(spec):] if WIN: # pragma: no cover subpath = subpath.replace('\\', '/') # windows - if cachebust: - subpath, kw = cachebust(subpath, kw) + # translate spec into overridden spec and lookup cache buster + # to modify subpath, kw if url is None: kw['subpath'] = subpath return request.route_url(route_name, **kw) @@ -2023,19 +2026,6 @@ class StaticURLInfo(object): # make sure it ends with a slash name = name + '/' - if config.registry.settings.get('pyramid.prevent_cachebust'): - cb = None - else: - cb = extra.pop('cachebust', None) - if cb: - def cachebust(subpath, kw): - subpath_tuple = tuple(subpath.split('/')) - subpath_tuple, kw = cb.pregenerate( - spec + subpath, subpath_tuple, kw) - return '/'.join(subpath_tuple), kw - else: - cachebust = None - if url_parse(name).netloc: # it's a URL # url, spec, route_name @@ -2044,14 +2034,11 @@ class StaticURLInfo(object): else: # it's a view name url = None - ten_years = 10 * 365 * 24 * 60 * 60 # more or less - default = ten_years if cb else None - cache_max_age = extra.pop('cache_max_age', default) + cache_max_age = extra.pop('cache_max_age', None) # create a view - cb_match = getattr(cb, 'match', None) view = static_view(spec, cache_max_age=cache_max_age, - use_subpath=True, cachebust_match=cb_match) + use_subpath=True) # Mutate extra to allow factory, etc to be passed through here. # Treat permission specially because we'd like to default to @@ -2083,7 +2070,7 @@ class StaticURLInfo(object): ) def register(): - registrations = self._get_registrations(config.registry) + registrations = self.registrations names = [t[0] for t in registrations] @@ -2092,7 +2079,7 @@ class StaticURLInfo(object): registrations.pop(idx) # url, spec, route_name - registrations.append((url, spec, route_name, cachebust)) + registrations.append((url, spec, route_name)) intr = config.introspectable('static views', name, @@ -2102,3 +2089,30 @@ class StaticURLInfo(object): intr['spec'] = spec config.action(None, callable=register, introspectables=(intr,)) + + def add_cache_buster(self, config, spec, cachebust): + def register(): + cache_busters = self.cache_busters + + specs = [t[0] for t in cache_busters] + if spec in specs: + idx = specs.index(spec) + cache_busters.pop(idx) + + lengths = [len(t[0]) for t in cache_busters] + new_idx = bisect.bisect_left(lengths, len(spec)) + cache_busters.insert(new_idx, (spec, cachebust)) + + intr = config.introspectable('cache busters', + spec, + 'cache buster for %r' % spec, + 'cache buster') + intr['cachebust'] = cachebust + intr['spec'] = spec + + config.action(None, callable=register, introspectables=(intr,)) + + def _find_cache_buster(self, registry, spec): + for base_spec, cachebust in self.cache_busters: + if base_spec.startswith(spec): + pass diff --git a/pyramid/interfaces.py b/pyramid/interfaces.py index 90534593c..bdf5bdfbe 100644 --- a/pyramid/interfaces.py +++ b/pyramid/interfaces.py @@ -584,6 +584,9 @@ class IStaticURLInfo(Interface): def generate(path, request, **kw): """ Generate a URL for the given path """ + def add_cache_buster(config, spec, cache_buster): + """ Add a new cache buster to a particular set of assets """ + class IResponseFactory(Interface): """ A utility which generates a response """ def __call__(request): @@ -1186,45 +1189,23 @@ class IPredicateList(Interface): class ICacheBuster(Interface): """ - Instances of ``ICacheBuster`` may be provided as arguments to - :meth:`~pyramid.config.Configurator.add_static_view`. Instances of - ``ICacheBuster`` provide mechanisms for generating a cache bust token for - a static asset, modifying a static asset URL to include a cache bust token, - and, optionally, unmodifying a static asset URL in order to look up an - asset. See :ref:`cache_busting`. + A cache buster modifies the URL generation machinery for + :meth:`~pyramid.request.Request.static_url`. See :ref:`cache_busting`. .. versionadded:: 1.6 """ - def pregenerate(pathspec, subpath, kw): + def __call__(pathspec, subpath, kw): """ Modifies a subpath and/or keyword arguments from which a static asset URL will be computed during URL generation. The ``pathspec`` argument is the path specification for the resource to be cache busted. - The ``subpath`` argument is a tuple of path elements that represent the - portion of the asset URL which is used to find the asset. The ``kw`` - argument is a dict of keywords that are to be passed eventually to - :meth:`~pyramid.request.Request.route_url` for URL generation. The - return value should be a two-tuple of ``(subpath, kw)`` which are - versions of the same arguments modified to include the cache bust token - in the generated URL. - """ - - def match(subpath): - """ - Performs the logical inverse of - :meth:`~pyramid.interfaces.ICacheBuster.pregenerate` by taking a - subpath from a cache busted URL and removing the cache bust token, so - that :app:`Pyramid` can find the underlying asset. - - ``subpath`` is the subpath portion of the URL for an incoming request - for a static asset. The return value should be the same tuple with the - cache busting token elided. - - If the cache busting scheme in use doesn't specifically modify the path - portion of the generated URL (e.g. it adds a query string), a method - which implements this interface may not be necessary. It is - permissible for an instance of - :class:`~pyramid.interfaces.ICacheBuster` to omit this method. + The ``subpath`` argument is a path of ``/``-delimited segments that + represent the portion of the asset URL which is used to find the asset. + The ``kw`` argument is a dict of keywords that are to be passed + eventually to :meth:`~pyramid.request.Request.static_url` for URL + generation. The return value should be a two-tuple of + ``(subpath, kw)`` which are versions of the same arguments modified + to include the cache bust token in the generated URL. """ # configuration phases: a lower phase number means the actions associated diff --git a/pyramid/static.py b/pyramid/static.py index c7a5c7ba5..cda98bea4 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -179,7 +179,7 @@ class QueryStringCacheBuster(object): def __init__(self, param='x'): self.param = param - def pregenerate(self, pathspec, subpath, kw): + def __call__(self, pathspec, subpath, kw): token = self.tokenize(pathspec) query = kw.setdefault('_query', {}) if isinstance(query, dict): @@ -289,8 +289,6 @@ class ManifestCacheBuster(object): self._mtime = mtime return self._manifest - def pregenerate(self, pathspec, subpath, kw): - path = '/'.join(subpath) - path = self.manifest.get(path, path) - new_subpath = path.split('/') - return (new_subpath, kw) + def __call__(self, pathspec, subpath, kw): + subpath = self.manifest.get(subpath, subpath) + return (subpath, kw) diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index acfb81962..020ed131d 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -3865,22 +3865,11 @@ class TestStaticURLInfo(unittest.TestCase): def _makeOne(self): return self._getTargetClass()() - def _makeConfig(self, registrations=None): - config = DummyConfig() - registry = DummyRegistry() - if registrations is not None: - registry._static_url_registrations = registrations - config.registry = registry - return config - def _makeRequest(self): request = DummyRequest() request.registry = DummyRegistry() return request - def _assertRegistrations(self, config, expected): - self.assertEqual(config.registry._static_url_registrations, expected) - def test_verifyClass(self): from pyramid.interfaces import IStaticURLInfo from zope.interface.verify import verifyClass @@ -4002,12 +3991,12 @@ class TestStaticURLInfo(unittest.TestCase): 'http://example.com/abc%20def#La%20Pe%C3%B1a') def test_generate_url_cachebust(self): - def cachebust(subpath, kw): + def cachebust(request, subpath, kw): kw['foo'] = 'bar' return 'foo' + '/' + subpath, kw inst = self._makeOne() - registrations = [(None, 'package:path/', '__viewname', cachebust)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [(None, 'package:path/', '__viewname', cachebust)] + inst.cache_busters = [('package:path/', cachebust)] request = self._makeRequest() def route_url(n, **kw): self.assertEqual(n, '__viewname') @@ -4016,88 +4005,88 @@ class TestStaticURLInfo(unittest.TestCase): inst.generate('package:path/abc', request) def test_add_already_exists(self): + config = DummyConfig() inst = self._makeOne() - config = self._makeConfig( - [('http://example.com/', 'package:path/', None)]) + inst.registrations = [('http://example.com/', 'package:path/', None)] inst.add(config, 'http://example.com', 'anotherpackage:path') expected = [ - ('http://example.com/', 'anotherpackage:path/', None, None)] - self._assertRegistrations(config, expected) + ('http://example.com/', 'anotherpackage:path/', None, None)] + self.assertEqual(inst.registrations, expected) def test_add_package_root(self): + config = DummyConfig() inst = self._makeOne() - config = self._makeConfig() inst.add(config, 'http://example.com', 'package:') - expected = [('http://example.com/', 'package:', None, None)] - self._assertRegistrations(config, expected) + expected = [('http://example.com/', 'package:', None, None)] + self.assertEqual(inst.registrations, expected) def test_add_url_withendslash(self): + config = DummyConfig() inst = self._makeOne() - config = self._makeConfig() inst.add(config, 'http://example.com/', 'anotherpackage:path') expected = [ ('http://example.com/', 'anotherpackage:path/', None, None)] - self._assertRegistrations(config, expected) + self.assertEqual(inst.registrations, expected) def test_add_url_noendslash(self): + config = DummyConfig() inst = self._makeOne() - config = self._makeConfig() inst.add(config, 'http://example.com', 'anotherpackage:path') expected = [ ('http://example.com/', 'anotherpackage:path/', None, None)] - self._assertRegistrations(config, expected) + self.assertEqual(inst.registrations, expected) def test_add_url_noscheme(self): + config = DummyConfig() inst = self._makeOne() - config = self._makeConfig() inst.add(config, '//example.com', 'anotherpackage:path') expected = [('//example.com/', 'anotherpackage:path/', None, None)] - self._assertRegistrations(config, expected) + self.assertEqual(inst.registrations, expected) def test_add_viewname(self): from pyramid.security import NO_PERMISSION_REQUIRED from pyramid.static import static_view - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1) expected = [(None, 'anotherpackage:path/', '__view/', None)] - self._assertRegistrations(config, expected) + self.assertEqual(inst.registrations, expected) self.assertEqual(config.route_args, ('__view/', 'view/*subpath')) self.assertEqual(config.view_kw['permission'], NO_PERMISSION_REQUIRED) self.assertEqual(config.view_kw['view'].__class__, static_view) def test_add_viewname_with_route_prefix(self): - config = self._makeConfig() + config = DummyConfig() config.route_prefix = '/abc' inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path',) expected = [(None, 'anotherpackage:path/', '__/abc/view/', None)] - self._assertRegistrations(config, expected) + self.assertEqual(inst.registrations, expected) self.assertEqual(config.route_args, ('__/abc/view/', 'view/*subpath')) def test_add_viewname_with_permission(self): - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1, permission='abc') self.assertEqual(config.view_kw['permission'], 'abc') def test_add_viewname_with_context(self): - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1, context=DummyContext) self.assertEqual(config.view_kw['context'], DummyContext) def test_add_viewname_with_for_(self): - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1, for_=DummyContext) self.assertEqual(config.view_kw['context'], DummyContext) def test_add_viewname_with_renderer(self): - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1, renderer='mypackage:templates/index.pt') @@ -4105,7 +4094,7 @@ class TestStaticURLInfo(unittest.TestCase): 'mypackage:templates/index.pt') def test_add_cachebust_prevented(self): - config = self._makeConfig() + config = DummyConfig() config.registry.settings['pyramid.prevent_cachebust'] = True inst = self._makeOne() inst.add(config, 'view', 'mypackage:path', cachebust=True) @@ -4113,7 +4102,7 @@ class TestStaticURLInfo(unittest.TestCase): self.assertEqual(cachebust, None) def test_add_cachebust_custom(self): - config = self._makeConfig() + config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'mypackage:path', cachebust=DummyCacheBuster('foo')) @@ -4236,7 +4225,8 @@ class DummyMultiView: class DummyCacheBuster(object): def __init__(self, token): self.token = token - def pregenerate(self, pathspec, subpath, kw): + + def __call__(self, pathspec, subpath, kw): kw['x'] = self.token return subpath, kw -- cgit v1.2.3 From acfdc7085b836045e41568731e5f3223f34d635d Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 4 Dec 2015 02:17:44 -0800 Subject: - add emphasize lines, minor grammar, rewrap 79 columns --- docs/narr/zca.rst | 257 +++++++++++++++++++++++++----------------------------- 1 file changed, 121 insertions(+), 136 deletions(-) diff --git a/docs/narr/zca.rst b/docs/narr/zca.rst index b0e9b1709..784886563 100644 --- a/docs/narr/zca.rst +++ b/docs/narr/zca.rst @@ -9,17 +9,16 @@ .. _zca_chapter: Using the Zope Component Architecture in :app:`Pyramid` -========================================================== +======================================================= -Under the hood, :app:`Pyramid` uses a :term:`Zope Component -Architecture` component registry as its :term:`application registry`. -The Zope Component Architecture is referred to colloquially as the -"ZCA." +Under the hood, :app:`Pyramid` uses a :term:`Zope Component Architecture` +component registry as its :term:`application registry`. The Zope Component +Architecture is referred to colloquially as the "ZCA." The ``zope.component`` API used to access data in a traditional Zope -application can be opaque. For example, here is a typical "unnamed -utility" lookup using the :func:`zope.component.getUtility` global API -as it might appear in a traditional Zope application: +application can be opaque. For example, here is a typical "unnamed utility" +lookup using the :func:`zope.component.getUtility` global API as it might +appear in a traditional Zope application: .. code-block:: python :linenos: @@ -28,23 +27,21 @@ as it might appear in a traditional Zope application: from zope.component import getUtility settings = getUtility(ISettings) -After this code runs, ``settings`` will be a Python dictionary. But -it's unlikely that any "civilian" will be able to figure this out just -by reading the code casually. When the ``zope.component.getUtility`` -API is used by a developer, the conceptual load on a casual reader of -code is high. +After this code runs, ``settings`` will be a Python dictionary. But it's +unlikely that any "civilian" will be able to figure this out just by reading +the code casually. When the ``zope.component.getUtility`` API is used by a +developer, the conceptual load on a casual reader of code is high. -While the ZCA is an excellent tool with which to build a *framework* -such as :app:`Pyramid`, it is not always the best tool with which -to build an *application* due to the opacity of the ``zope.component`` -APIs. Accordingly, :app:`Pyramid` tends to hide the presence of the -ZCA from application developers. You needn't understand the ZCA to -create a :app:`Pyramid` application; its use is effectively only a -framework implementation detail. +While the ZCA is an excellent tool with which to build a *framework* such as +:app:`Pyramid`, it is not always the best tool with which to build an +*application* due to the opacity of the ``zope.component`` APIs. Accordingly, +:app:`Pyramid` tends to hide the presence of the ZCA from application +developers. You needn't understand the ZCA to create a :app:`Pyramid` +application; its use is effectively only a framework implementation detail. -However, developers who are already used to writing :term:`Zope` -applications often still wish to use the ZCA while building a -:app:`Pyramid` application; :app:`Pyramid` makes this possible. +However, developers who are already used to writing :term:`Zope` applications +often still wish to use the ZCA while building a :app:`Pyramid` application. +:app:`Pyramid` makes this possible. .. index:: single: get_current_registry @@ -52,87 +49,81 @@ applications often still wish to use the ZCA while building a single: getSiteManager single: ZCA global API -Using the ZCA Global API in a :app:`Pyramid` Application ------------------------------------------------------------ - -:term:`Zope` uses a single ZCA registry -- the "global" ZCA registry --- for all Zope applications that run in the same Python process, -effectively making it impossible to run more than one Zope application -in a single process. - -However, for ease of deployment, it's often useful to be able to run more -than a single application per process. For example, use of a -:term:`PasteDeploy` "composite" allows you to run separate individual WSGI -applications in the same process, each answering requests for some URL -prefix. This makes it possible to run, for example, a TurboGears application -at ``/turbogears`` and a :app:`Pyramid` application at ``/pyramid``, both -served up using the same :term:`WSGI` server within a single Python process. - -Most production Zope applications are relatively large, making it -impractical due to memory constraints to run more than one Zope -application per Python process. However, a :app:`Pyramid` application -may be very small and consume very little memory, so it's a reasonable -goal to be able to run more than one :app:`Pyramid` application per -process. - -In order to make it possible to run more than one :app:`Pyramid` -application in a single process, :app:`Pyramid` defaults to using a -separate ZCA registry *per application*. - -While this services a reasonable goal, it causes some issues when -trying to use patterns which you might use to build a typical -:term:`Zope` application to build a :app:`Pyramid` application. -Without special help, ZCA "global" APIs such as -:func:`zope.component.getUtility` and :func:`zope.component.getSiteManager` -will use the ZCA "global" registry. Therefore, these APIs -will appear to fail when used in a :app:`Pyramid` application, -because they'll be consulting the ZCA global registry rather than the -component registry associated with your :app:`Pyramid` application. - -There are three ways to fix this: by disusing the ZCA global API -entirely, by using -:meth:`pyramid.config.Configurator.hook_zca` or by passing -the ZCA global registry to the :term:`Configurator` constructor at -startup time. We'll describe all three methods in this section. +Using the ZCA global API in a :app:`Pyramid` application +-------------------------------------------------------- + +:term:`Zope` uses a single ZCA registry—the "global" ZCA registry—for all Zope +applications that run in the same Python process, effectively making it +impossible to run more than one Zope application in a single process. + +However, for ease of deployment, it's often useful to be able to run more than +a single application per process. For example, use of a :term:`PasteDeploy` +"composite" allows you to run separate individual WSGI applications in the same +process, each answering requests for some URL prefix. This makes it possible +to run, for example, a TurboGears application at ``/turbogears`` and a +:app:`Pyramid` application at ``/pyramid``, both served up using the same +:term:`WSGI` server within a single Python process. + +Most production Zope applications are relatively large, making it impractical +due to memory constraints to run more than one Zope application per Python +process. However, a :app:`Pyramid` application may be very small and consume +very little memory, so it's a reasonable goal to be able to run more than one +:app:`Pyramid` application per process. + +In order to make it possible to run more than one :app:`Pyramid` application in +a single process, :app:`Pyramid` defaults to using a separate ZCA registry *per +application*. + +While this services a reasonable goal, it causes some issues when trying to use +patterns which you might use to build a typical :term:`Zope` application to +build a :app:`Pyramid` application. Without special help, ZCA "global" APIs +such as :func:`zope.component.getUtility` and +:func:`zope.component.getSiteManager` will use the ZCA "global" registry. +Therefore, these APIs will appear to fail when used in a :app:`Pyramid` +application, because they'll be consulting the ZCA global registry rather than +the component registry associated with your :app:`Pyramid` application. + +There are three ways to fix this: by disusing the ZCA global API entirely, by +using :meth:`pyramid.config.Configurator.hook_zca` or by passing the ZCA global +registry to the :term:`Configurator` constructor at startup time. We'll +describe all three methods in this section. .. index:: single: request.registry .. _disusing_the_global_zca_api: -Disusing the Global ZCA API +Disusing the global ZCA API +++++++++++++++++++++++++++ ZCA "global" API functions such as ``zope.component.getSiteManager``, ``zope.component.getUtility``, :func:`zope.component.getAdapter`, and :func:`zope.component.getMultiAdapter` aren't strictly necessary. Every -component registry has a method API that offers the same -functionality; it can be used instead. For example, presuming the -``registry`` value below is a Zope Component Architecture component -registry, the following bit of code is equivalent to -``zope.component.getUtility(IFoo)``: +component registry has a method API that offers the same functionality; it can +be used instead. For example, presuming the ``registry`` value below is a Zope +Component Architecture component registry, the following bit of code is +equivalent to ``zope.component.getUtility(IFoo)``: .. code-block:: python registry.getUtility(IFoo) -The full method API is documented in the ``zope.component`` package, -but it largely mirrors the "global" API almost exactly. +The full method API is documented in the ``zope.component`` package, but it +largely mirrors the "global" API almost exactly. -If you are willing to disuse the "global" ZCA APIs and use the method -interface of a registry instead, you need only know how to obtain the -:app:`Pyramid` component registry. +If you are willing to disuse the "global" ZCA APIs and use the method interface +of a registry instead, you need only know how to obtain the :app:`Pyramid` +component registry. There are two ways of doing so: -- use the :func:`pyramid.threadlocal.get_current_registry` - function within :app:`Pyramid` view or resource code. This will - always return the "current" :app:`Pyramid` application registry. +- use the :func:`pyramid.threadlocal.get_current_registry` function within + :app:`Pyramid` view or resource code. This will always return the "current" + :app:`Pyramid` application registry. -- use the attribute of the :term:`request` object named ``registry`` - in your :app:`Pyramid` view code, eg. ``request.registry``. This - is the ZCA component registry related to the running - :app:`Pyramid` application. +- use the attribute of the :term:`request` object named ``registry`` in your + :app:`Pyramid` view code, e.g., ``request.registry``. This is the ZCA + component registry related to the running :app:`Pyramid` application. See :ref:`threadlocals_chapter` for more information about :func:`pyramid.threadlocal.get_current_registry`. @@ -142,7 +133,7 @@ See :ref:`threadlocals_chapter` for more information about .. _hook_zca: -Enabling the ZCA Global API by Using ``hook_zca`` +Enabling the ZCA global API by using ``hook_zca`` +++++++++++++++++++++++++++++++++++++++++++++++++ Consider the following bit of idiomatic :app:`Pyramid` startup code: @@ -157,34 +148,31 @@ Consider the following bit of idiomatic :app:`Pyramid` startup code: config.include('some.other.package') return config.make_wsgi_app() -When the ``app`` function above is run, a :term:`Configurator` is -constructed. When the configurator is created, it creates a *new* -:term:`application registry` (a ZCA component registry). A new -registry is constructed whenever the ``registry`` argument is omitted -when a :term:`Configurator` constructor is called, or when a -``registry`` argument with a value of ``None`` is passed to a -:term:`Configurator` constructor. - -During a request, the application registry created by the Configurator -is "made current". This means calls to -:func:`~pyramid.threadlocal.get_current_registry` in the thread -handling the request will return the component registry associated -with the application. - -As a result, application developers can use ``get_current_registry`` -to get the registry and thus get access to utilities and such, as per -:ref:`disusing_the_global_zca_api`. But they still cannot use the -global ZCA API. Without special treatment, the ZCA global APIs will -always return the global ZCA registry (the one in -``zope.component.globalregistry.base``). - -To "fix" this and make the ZCA global APIs use the "current" -:app:`Pyramid` registry, you need to call -:meth:`~pyramid.config.Configurator.hook_zca` within your setup code. -For example: +When the ``app`` function above is run, a :term:`Configurator` is constructed. +When the configurator is created, it creates a *new* :term:`application +registry` (a ZCA component registry). A new registry is constructed whenever +the ``registry`` argument is omitted, when a :term:`Configurator` constructor +is called, or when a ``registry`` argument with a value of ``None`` is passed +to a :term:`Configurator` constructor. + +During a request, the application registry created by the Configurator is "made +current". This means calls to +:func:`~pyramid.threadlocal.get_current_registry` in the thread handling the +request will return the component registry associated with the application. + +As a result, application developers can use ``get_current_registry`` to get the +registry and thus get access to utilities and such, as per +:ref:`disusing_the_global_zca_api`. But they still cannot use the global ZCA +API. Without special treatment, the ZCA global APIs will always return the +global ZCA registry (the one in ``zope.component.globalregistry.base``). + +To "fix" this and make the ZCA global APIs use the "current" :app:`Pyramid` +registry, you need to call :meth:`~pyramid.config.Configurator.hook_zca` within +your setup code. For example: .. code-block:: python :linenos: + :emphasize-lines: 5 from pyramid.config import Configurator @@ -194,9 +182,9 @@ For example: config.include('some.other.application') return config.make_wsgi_app() -We've added a line to our original startup code, line number 6, which -calls ``config.hook_zca()``. The effect of this line under the hood -is that an analogue of the following code is executed: +We've added a line to our original startup code, line number 5, which calls +``config.hook_zca()``. The effect of this line under the hood is that an +analogue of the following code is executed: .. code-block:: python :linenos: @@ -205,17 +193,15 @@ is that an analogue of the following code is executed: from pyramid.threadlocal import get_current_registry getSiteManager.sethook(get_current_registry) -This causes the ZCA global API to start using the :app:`Pyramid` -application registry in threads which are running a :app:`Pyramid` -request. +This causes the ZCA global API to start using the :app:`Pyramid` application +registry in threads which are running a :app:`Pyramid` request. -Calling ``hook_zca`` is usually sufficient to "fix" the problem of -being able to use the global ZCA API within a :app:`Pyramid` -application. However, it also means that a Zope application that is -running in the same process may start using the :app:`Pyramid` -global registry instead of the Zope global registry, effectively -inverting the original problem. In such a case, follow the steps in -the next section, :ref:`using_the_zca_global_registry`. +Calling ``hook_zca`` is usually sufficient to "fix" the problem of being able +to use the global ZCA API within a :app:`Pyramid` application. However, it +also means that a Zope application that is running in the same process may +start using the :app:`Pyramid` global registry instead of the Zope global +registry, effectively inverting the original problem. In such a case, follow +the steps in the next section, :ref:`using_the_zca_global_registry`. .. index:: single: get_current_registry @@ -224,14 +210,15 @@ the next section, :ref:`using_the_zca_global_registry`. .. _using_the_zca_global_registry: -Enabling the ZCA Global API by Using The ZCA Global Registry +Enabling the ZCA global API by using the ZCA global registry ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -You can tell your :app:`Pyramid` application to use the ZCA global -registry at startup time instead of constructing a new one: +You can tell your :app:`Pyramid` application to use the ZCA global registry at +startup time instead of constructing a new one: .. code-block:: python :linenos: + :emphasize-lines: 5-7 from zope.component import getGlobalSiteManager from pyramid.config import Configurator @@ -243,16 +230,14 @@ registry at startup time instead of constructing a new one: config.include('some.other.application') return config.make_wsgi_app() -Lines 5, 6, and 7 above are the interesting ones. Line 5 retrieves -the global ZCA component registry. Line 6 creates a -:term:`Configurator`, passing the global ZCA registry into its -constructor as the ``registry`` argument. Line 7 "sets up" the global -registry with Pyramid-specific registrations; this is code that is -normally executed when a registry is constructed rather than created, +Lines 5, 6, and 7 above are the interesting ones. Line 5 retrieves the global +ZCA component registry. Line 6 creates a :term:`Configurator`, passing the +global ZCA registry into its constructor as the ``registry`` argument. Line 7 +"sets up" the global registry with Pyramid-specific registrations; this is code +that is normally executed when a registry is constructed rather than created, but we must call it "by hand" when we pass an explicit registry. -At this point, :app:`Pyramid` will use the ZCA global registry -rather than creating a new application-specific registry; since by -default the ZCA global API will use this registry, things will work as -you might expect a Zope app to when you use the global ZCA API. - +At this point, :app:`Pyramid` will use the ZCA global registry rather than +creating a new application-specific registry. Since by default the ZCA global +API will use this registry, things will work as you might expect in a Zope app +when you use the global ZCA API. -- cgit v1.2.3 From a26e3298ddd73ad782132f9b1098e02f7ed55c42 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 4 Dec 2015 02:39:39 -0800 Subject: update references to references to python-distribute.org (see #2141 for discussion) --- docs/narr/project.rst | 4 ++-- docs/narr/scaffolding.rst | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/narr/project.rst b/docs/narr/project.rst index 25f3931e9..ec40bc67c 100644 --- a/docs/narr/project.rst +++ b/docs/narr/project.rst @@ -680,8 +680,8 @@ testing, packaging, and distributing your application. ``setup.py`` is the de facto standard which Python developers use to distribute their reusable code. You can read more about ``setup.py`` files and their usage in the `Setuptools documentation - `_ and `The Hitchhiker's - Guide to Packaging `_. + `_ and `Python Packaging + User Guide `_. Our generated ``setup.py`` looks like this: diff --git a/docs/narr/scaffolding.rst b/docs/narr/scaffolding.rst index 8677359de..164ceb3bf 100644 --- a/docs/narr/scaffolding.rst +++ b/docs/narr/scaffolding.rst @@ -22,10 +22,10 @@ found by the ``pcreate`` command. To create a scaffold template, create a Python :term:`distribution` to house the scaffold which includes a ``setup.py`` that relies on the ``setuptools`` -package. See `Creating a Package -`_ for more information about -how to do this. For example, we'll pretend the distribution you create is -named ``CoolExtension``, and it has a package directory within it named +package. See `Packaging and Distributing Projects +`_ for more information +about how to do this. For example, we'll pretend the distribution you create +is named ``CoolExtension``, and it has a package directory within it named ``coolextension``. Once you've created the distribution, put a "scaffolds" directory within your -- cgit v1.2.3 From 1fefda3e6d104e6a308c2daa898a85116f4a3794 Mon Sep 17 00:00:00 2001 From: Ismail Date: Mon, 7 Dec 2015 14:09:11 +0000 Subject: Fix minor typo --- pyramid/config/views.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index e386bc4e1..a6899abbf 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1809,7 +1809,7 @@ class ViewsConfiguratorMixin(object): signatures than the ones supported by :app:`Pyramid` as described in its narrative documentation. - The ``mapper`` should argument be an object implementing + The ``mapper`` argument should be an object implementing :class:`pyramid.interfaces.IViewMapperFactory` or a :term:`dotted Python name` to such an object. The provided ``mapper`` will become the default view mapper to be used by all subsequent :term:`view -- cgit v1.2.3 From 62222d69b7b6ef573d7f52529b15285af4111f20 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 7 Dec 2015 21:11:11 -0600 Subject: support getting the file path from a FSAssetSource even if it doesn't exist --- pyramid/config/assets.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/pyramid/config/assets.py b/pyramid/config/assets.py index bbdf18ced..d05314384 100644 --- a/pyramid/config/assets.py +++ b/pyramid/config/assets.py @@ -262,12 +262,15 @@ class FSAssetSource(object): def __init__(self, prefix): self.prefix = prefix - def get_filename(self, resource_name): + def get_path(self, resource_name): if resource_name: path = os.path.join(self.prefix, resource_name) else: path = self.prefix + return path + def get_filename(self, resource_name): + path = self.get_path(resource_name) if os.path.exists(path): return path -- cgit v1.2.3 From d0bd5fb326d8999e3a40e6e2d121aa69cfe05476 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 7 Dec 2015 22:58:06 -0600 Subject: add a first cut at an add_cache_buster api --- pyramid/config/views.py | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 67a70145c..f496dfb7d 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1,5 +1,5 @@ -import bisect import inspect +import posixpath import operator import os import warnings @@ -21,6 +21,7 @@ from pyramid.interfaces import ( IException, IExceptionViewClassifier, IMultiView, + IPackageOverrides, IRendererFactory, IRequest, IResponse, @@ -1987,6 +1988,8 @@ class StaticURLInfo(object): subpath = subpath.replace('\\', '/') # windows # translate spec into overridden spec and lookup cache buster # to modify subpath, kw + subpath, kw = self._bust_asset_path( + request.registry, spec, subpath, kw) if url is None: kw['subpath'] = subpath return request.route_url(route_name, **kw) @@ -2099,9 +2102,7 @@ class StaticURLInfo(object): idx = specs.index(spec) cache_busters.pop(idx) - lengths = [len(t[0]) for t in cache_busters] - new_idx = bisect.bisect_left(lengths, len(spec)) - cache_busters.insert(new_idx, (spec, cachebust)) + cache_busters.insert(0, (spec, cachebust)) intr = config.introspectable('cache busters', spec, @@ -2112,7 +2113,24 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) - def _find_cache_buster(self, registry, spec): + def _bust_asset_path(self, registry, spec, subpath, kw): + pkg_name, pkg_subpath = spec.split(':') + absspec = rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) + overrides = registry.queryUtility(IPackageOverrides, name=pkg_name) + if overrides is not None: + resource_name = posixpath.join(pkg_subpath, subpath) + sources = overrides.filtered_sources(resource_name) + for source, filtered_path in sources: + rawspec = source.get_path(filtered_path) + if hasattr(source, 'pkg_name'): + rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) + break + for base_spec, cachebust in self.cache_busters: - if base_spec.startswith(spec): - pass + if ( + base_spec == rawspec or + (base_spec.endswith('/') and rawspec.startswith(base_spec)) + ): + subpath, kw = cachebust(absspec, subpath, kw) + break + return subpath, kw -- cgit v1.2.3 From 73630eae045549b792c4e3ef77920357f89c6874 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 7 Dec 2015 23:16:26 -0600 Subject: sort by length such that longer paths are tested first --- pyramid/config/views.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index f496dfb7d..16b150a9e 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1,3 +1,4 @@ +import bisect import inspect import posixpath import operator @@ -2102,7 +2103,9 @@ class StaticURLInfo(object): idx = specs.index(spec) cache_busters.pop(idx) - cache_busters.insert(0, (spec, cachebust)) + lengths = [len(t[0]) for t in cache_busters] + new_idx = bisect.bisect_left(lengths, len(spec)) + cache_busters.insert(new_idx, (spec, cachebust)) intr = config.introspectable('cache busters', spec, @@ -2126,7 +2129,7 @@ class StaticURLInfo(object): rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) break - for base_spec, cachebust in self.cache_busters: + for base_spec, cachebust in reversed(self.cache_busters): if ( base_spec == rawspec or (base_spec.endswith('/') and rawspec.startswith(base_spec)) -- cgit v1.2.3 From 54d00fd7ff0fcfd19799cbffedb860d08604b83c Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 7 Dec 2015 23:21:22 -0600 Subject: support os.sep on windows --- pyramid/config/views.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 16b150a9e..304ce2d43 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -2132,7 +2132,11 @@ class StaticURLInfo(object): for base_spec, cachebust in reversed(self.cache_busters): if ( base_spec == rawspec or - (base_spec.endswith('/') and rawspec.startswith(base_spec)) + ( + base_spec.endswith(os.sep) + if os.path.isabs(base_spec) + else base_spec.endswith('/') + ) and rawspec.startswith(base_spec) ): subpath, kw = cachebust(absspec, subpath, kw) break -- cgit v1.2.3 From d4177a51aff1a46d1c2223db6ed7afd99964e8ad Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Tue, 8 Dec 2015 01:41:06 -0800 Subject: update narrative docs and literalinclude source files that use the starter scaffold to reflect its current state --- docs/narr/MyProject/development.ini | 6 +- .../MyProject/myproject/templates/mytemplate.pt | 13 +-- docs/narr/MyProject/myproject/tests.py | 1 + docs/narr/MyProject/production.ini | 2 +- docs/narr/MyProject/setup.py | 45 ++++------- docs/narr/project.rst | 94 ++++++++++++---------- 6 files changed, 81 insertions(+), 80 deletions(-) diff --git a/docs/narr/MyProject/development.ini b/docs/narr/MyProject/development.ini index a9a26e56b..749e574eb 100644 --- a/docs/narr/MyProject/development.ini +++ b/docs/narr/MyProject/development.ini @@ -11,7 +11,7 @@ pyramid.debug_authorization = false pyramid.debug_notfound = false pyramid.debug_routematch = false pyramid.default_locale_name = en -pyramid.includes = +pyramid.includes = pyramid_debugtoolbar # By default, the toolbar only appears for clients from IP addresses @@ -24,7 +24,7 @@ pyramid.includes = [server:main] use = egg:waitress#main -host = 0.0.0.0 +host = 127.0.0.1 port = 6543 ### @@ -57,4 +57,4 @@ level = NOTSET formatter = generic [formatter_generic] -format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s +format = %(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s diff --git a/docs/narr/MyProject/myproject/templates/mytemplate.pt b/docs/narr/MyProject/myproject/templates/mytemplate.pt index e6b00a145..65d7f0609 100644 --- a/docs/narr/MyProject/myproject/templates/mytemplate.pt +++ b/docs/narr/MyProject/myproject/templates/mytemplate.pt @@ -8,12 +8,12 @@ - Starter Template for The Pyramid Web Framework + Starter Scaffold for The Pyramid Web Framework - + @@ -33,19 +33,20 @@
-

Pyramid starter template

-

Welcome to ${project}, an application generated by
the Pyramid Web Framework.

+

Pyramid Starter scaffold

+

Welcome to ${project}, an application generated by
the Pyramid Web Framework 1.6b2.

diff --git a/docs/narr/MyProject/myproject/tests.py b/docs/narr/MyProject/myproject/tests.py index 8c60407e5..63d00910c 100644 --- a/docs/narr/MyProject/myproject/tests.py +++ b/docs/narr/MyProject/myproject/tests.py @@ -16,6 +16,7 @@ class ViewTests(unittest.TestCase): info = my_view(request) self.assertEqual(info['project'], 'MyProject') + class ViewIntegrationTests(unittest.TestCase): def setUp(self): """ This sets up the application registry with the diff --git a/docs/narr/MyProject/production.ini b/docs/narr/MyProject/production.ini index 9eae9e03f..3ccbe6619 100644 --- a/docs/narr/MyProject/production.ini +++ b/docs/narr/MyProject/production.ini @@ -51,4 +51,4 @@ level = NOTSET formatter = generic [formatter_generic] -format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s +format = %(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s diff --git a/docs/narr/MyProject/setup.py b/docs/narr/MyProject/setup.py index 9f34540a7..8c019af51 100644 --- a/docs/narr/MyProject/setup.py +++ b/docs/narr/MyProject/setup.py @@ -1,42 +1,30 @@ -"""Setup for the MyProject package. - -""" import os -from setuptools import setup, find_packages - - -HERE = os.path.abspath(os.path.dirname(__file__)) - -with open(os.path.join(HERE, 'README.txt')) as fp: - README = fp.read() - - -with open(os.path.join(HERE, 'CHANGES.txt')) as fp: - CHANGES = fp.read() +from setuptools import setup, find_packages +here = os.path.abspath(os.path.dirname(__file__)) +with open(os.path.join(here, 'README.txt')) as f: + README = f.read() +with open(os.path.join(here, 'CHANGES.txt')) as f: + CHANGES = f.read() -REQUIRES = [ +requires = [ 'pyramid', 'pyramid_chameleon', 'pyramid_debugtoolbar', 'waitress', ] -TESTS_REQUIRE = [ - 'webtest' - ] - setup(name='MyProject', version='0.0', description='MyProject', long_description=README + '\n\n' + CHANGES, classifiers=[ - 'Programming Language :: Python', - 'Framework :: Pyramid', - 'Topic :: Internet :: WWW/HTTP', - 'Topic :: Internet :: WWW/HTTP :: WSGI :: Application', - ], + "Programming Language :: Python", + "Framework :: Pyramid", + "Topic :: Internet :: WWW/HTTP", + "Topic :: Internet :: WWW/HTTP :: WSGI :: Application", + ], author='', author_email='', url='', @@ -44,10 +32,11 @@ setup(name='MyProject', packages=find_packages(), include_package_data=True, zip_safe=False, - install_requires=REQUIRES, - tests_require=TESTS_REQUIRE, - test_suite='myproject', + install_requires=requires, + tests_require=requires, + test_suite="myproject", entry_points="""\ [paste.app_factory] main = myproject:main - """) + """, + ) diff --git a/docs/narr/project.rst b/docs/narr/project.rst index ec40bc67c..4785b60c4 100644 --- a/docs/narr/project.rst +++ b/docs/narr/project.rst @@ -77,7 +77,7 @@ The below example uses the ``pcreate`` command to create a project with the On UNIX: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/pcreate -s starter MyProject @@ -90,7 +90,7 @@ Or on Windows: Here's sample output from a run of ``pcreate`` on UNIX for a project we name ``MyProject``: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/pcreate -s starter MyProject Creating template pyramid @@ -158,7 +158,7 @@ created project directory. On UNIX: -.. code-block:: text +.. code-block:: bash $ cd MyProject $ $VENV/bin/python setup.py develop @@ -172,7 +172,7 @@ Or on Windows: Elided output from a run of this command on UNIX is shown below: -.. code-block:: text +.. code-block:: bash $ cd MyProject $ $VENV/bin/python setup.py develop @@ -198,7 +198,7 @@ directory of your virtualenv). On UNIX: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/python setup.py test -q @@ -210,7 +210,7 @@ Or on Windows: Here's sample output from a test run on UNIX: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/python setup.py test -q running test @@ -221,11 +221,23 @@ Here's sample output from a test run on UNIX: writing dependency_links to MyProject.egg-info/dependency_links.txt writing entry points to MyProject.egg-info/entry_points.txt reading manifest file 'MyProject.egg-info/SOURCES.txt' + reading manifest template 'MANIFEST.in' + warning: no files found matching '*.cfg' + warning: no files found matching '*.rst' + warning: no files found matching '*.ico' under directory 'myproject' + warning: no files found matching '*.gif' under directory 'myproject' + warning: no files found matching '*.jpg' under directory 'myproject' + warning: no files found matching '*.txt' under directory 'myproject' + warning: no files found matching '*.mak' under directory 'myproject' + warning: no files found matching '*.mako' under directory 'myproject' + warning: no files found matching '*.js' under directory 'myproject' + warning: no files found matching '*.html' under directory 'myproject' + warning: no files found matching '*.xml' under directory 'myproject' writing manifest file 'MyProject.egg-info/SOURCES.txt' running build_ext - .. + . ---------------------------------------------------------------------- - Ran 1 test in 0.108s + Ran 1 test in 0.008s OK @@ -256,7 +268,7 @@ file. In our case, this file is named ``development.ini``. On UNIX: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/pserve development.ini @@ -268,38 +280,38 @@ On Windows: Here's sample output from a run of ``pserve`` on UNIX: -.. code-block:: text +.. code-block:: bash $ $VENV/bin/pserve development.ini - Starting server in PID 16601. - serving on http://0.0.0.0:6543 - -When you use ``pserve`` to start the application implied by the default -rendering of a scaffold, it will respond to requests on *all* IP addresses -possessed by your system, not just requests to ``localhost``. This is what the -``0.0.0.0`` in ``serving on http://0.0.0.0:6543`` means. The server will -respond to requests made to ``127.0.0.1`` and on any external IP address. For -example, your system might be configured to have an external IP address -``192.168.1.50``. If that's the case, if you use a browser running on the same -system as Pyramid, it will be able to access the application via -``http://127.0.0.1:6543/`` as well as via ``http://192.168.1.50:6543/``. -However, *other people* on other computers on the same network will also be -able to visit your Pyramid application in their browser by visiting -``http://192.168.1.50:6543/``. - -If you want to restrict access such that only a browser running on the same -machine as Pyramid will be able to access your Pyramid application, edit the + Starting server in PID 16208. + serving on http://127.0.0.1:6543 + +Access is restricted such that only a browser running on the same machine as +Pyramid will be able to access your Pyramid application. However, if you want +to open access to other machines on the same network, then edit the ``development.ini`` file, and replace the ``host`` value in the -``[server:main]`` section. Change it from ``0.0.0.0`` to ``127.0.0.1``. For +``[server:main]`` section, changing it from ``127.0.0.1`` to ``0.0.0.0``. For example: .. code-block:: ini [server:main] use = egg:waitress#main - host = 127.0.0.1 + host = 0.0.0.0 port = 6543 +Now when you use ``pserve`` to start the application, it will respond to +requests on *all* IP addresses possessed by your system, not just requests to +``localhost``. This is what the ``0.0.0.0`` in +``serving on http://0.0.0.0:6543`` means. The server will respond to requests +made to ``127.0.0.1`` and on any external IP address. For example, your system +might be configured to have an external IP address ``192.168.1.50``. If that's +the case, if you use a browser running on the same system as Pyramid, it will +be able to access the application via ``http://127.0.0.1:6543/`` as well as via +``http://192.168.1.50:6543/``. However, *other people* on other computers on +the same network will also be able to visit your Pyramid application in their +browser by visiting ``http://192.168.1.50:6543/``. + You can change the port on which the server runs on by changing the same portion of the ``development.ini`` file. For example, you can change the ``port = 6543`` line in the ``development.ini`` file's ``[server:main]`` @@ -347,7 +359,7 @@ For example, on UNIX: $ $VENV/bin/pserve development.ini --reload Starting subprocess with file monitor Starting server in PID 16601. - serving on http://0.0.0.0:6543 + serving on http://127.0.0.1:6543 Now if you make a change to any of your project's ``.py`` files or ``.ini`` files, you'll see the server restart automatically: @@ -357,7 +369,7 @@ files, you'll see the server restart automatically: development.ini changed; reloading... -------------------- Restarting -------------------- Starting server in PID 16602. - serving on http://0.0.0.0:6543 + serving on http://127.0.0.1:6543 Changes to template files (such as ``.pt`` or ``.mak`` files) won't cause the server to restart. Changes to template files don't require a server restart as @@ -579,18 +591,16 @@ file. The name ``main`` is a convention used by PasteDeploy signifying that it is the default application. The ``[server:main]`` section of the configuration file configures a WSGI -server which listens on TCP port 6543. It is configured to listen on all -interfaces (``0.0.0.0``). This means that any remote system which has TCP -access to your system can see your Pyramid application. +server which listens on TCP port 6543. It is configured to listen on localhost +only (``127.0.0.1``). .. _MyProject_ini_logging: -The sections that live between the markers ``# Begin logging configuration`` -and ``# End logging configuration`` represent Python's standard library -:mod:`logging` module configuration for your application. The sections between -these two markers are passed to the `logging module's config file configuration -engine `_ when -the ``pserve`` or ``pshell`` commands are executed. The default configuration +The sections after ``# logging configuration`` represent Python's standard +library :mod:`logging` module configuration for your application. These +sections are passed to the `logging module's config file configuration engine +`_ when the +``pserve`` or ``pshell`` commands are executed. The default configuration sends application logging output to the standard error output of your terminal. For more information about logging configuration, see :ref:`logging_chapter`. @@ -912,7 +922,7 @@ The ``tests.py`` module includes unit tests for your application. .. literalinclude:: MyProject/myproject/tests.py :language: python - :lines: 1-18 + :lines: 1-17 :linenos: This sample ``tests.py`` file has a single unit test defined within it. This -- cgit v1.2.3 From bf8014ec3458b46c427706988a8ca26380707cd7 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Tue, 8 Dec 2015 02:43:14 -0800 Subject: update narrative docs and literalinclude source files that use the starter scaffold to reflect its current state --- docs/narr/startup.rst | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/docs/narr/startup.rst b/docs/narr/startup.rst index 485f6b181..3e168eaea 100644 --- a/docs/narr/startup.rst +++ b/docs/narr/startup.rst @@ -6,15 +6,15 @@ Startup When you cause a :app:`Pyramid` application to start up in a console window, you'll see something much like this show up on the console: -.. code-block:: text +.. code-block:: bash - $ pserve development.ini - Starting server in PID 16601. - serving on 0.0.0.0:6543 view at http://127.0.0.1:6543 + $ $VENV/bin/pserve development.ini + Starting server in PID 16305. + serving on http://127.0.0.1:6543 This chapter explains what happens between the time you press the "Return" key on your keyboard after typing ``pserve development.ini`` and the time the line -``serving on 0.0.0.0:6543 ...`` is output to your console. +``serving on http://127.0.0.1:6543`` is output to your console. .. index:: single: startup process @@ -92,11 +92,11 @@ Here's a high-level time-ordered overview of what happens when you press In this case, the ``myproject.__init__:main`` function referred to by the entry point URI ``egg:MyProject`` (see :ref:`MyProject_ini` for more information about entry point URIs, and how they relate to callables) will - receive the key/value pairs ``{'pyramid.reload_templates':'true', - 'pyramid.debug_authorization':'false', 'pyramid.debug_notfound':'false', - 'pyramid.debug_routematch':'false', 'pyramid.debug_templates':'true', - 'pyramid.default_locale_name':'en'}``. See :ref:`environment_chapter` for - the meanings of these keys. + receive the key/value pairs ``{pyramid.reload_templates = true, + pyramid.debug_authorization = false, pyramid.debug_notfound = false, + pyramid.debug_routematch = false, pyramid.default_locale_name = en, and + pyramid.includes = pyramid_debugtoolbar}``. See :ref:`environment_chapter` + for the meanings of these keys. #. The ``main`` function first constructs a :class:`~pyramid.config.Configurator` instance, passing the ``settings`` @@ -131,10 +131,9 @@ Here's a high-level time-ordered overview of what happens when you press #. ``pserve`` starts the WSGI *server* defined within the ``[server:main]`` section. In our case, this is the Waitress server (``use = egg:waitress#main``), and it will listen on all interfaces (``host = - 0.0.0.0``), on port number 6543 (``port = 6543``). The server code itself - is what prints ``serving on 0.0.0.0:6543 view at http://127.0.0.1:6543``. - The server serves the application, and the application is running, waiting - to receive requests. + 127.0.0.1``), on port number 6543 (``port = 6543``). The server code itself + is what prints ``serving on http://127.0.0.1:6543``. The server serves the + application, and the application is running, waiting to receive requests. .. seealso:: Logging configuration is described in the :ref:`logging_chapter` chapter. -- cgit v1.2.3 From aecb4722640bc49a8e479f5eb5f332346535be8d Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 09:22:35 -0600 Subject: allow disabling the cache buster --- pyramid/config/views.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 304ce2d43..1115ccffc 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1982,15 +1982,16 @@ class StaticURLInfo(object): self.cache_busters = [] def generate(self, path, request, **kw): + disable_cache_buster = ( + request.registry.settings['pyramid.prevent_cachebust']) for (url, spec, route_name) in self.registrations: if path.startswith(spec): subpath = path[len(spec):] if WIN: # pragma: no cover subpath = subpath.replace('\\', '/') # windows - # translate spec into overridden spec and lookup cache buster - # to modify subpath, kw - subpath, kw = self._bust_asset_path( - request.registry, spec, subpath, kw) + if not disable_cache_buster: + subpath, kw = self._bust_asset_path( + request.registry, spec, subpath, kw) if url is None: kw['subpath'] = subpath return request.route_url(route_name, **kw) -- cgit v1.2.3 From eedef93f0c4c52ea11320bcd49386262fa7293a1 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 11:12:38 -0600 Subject: update the cache busting narrative to use add_cache_buster --- docs/narr/assets.rst | 103 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 72 insertions(+), 31 deletions(-) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 0f819570c..8b41f9b8a 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -366,8 +366,7 @@ resource and requests the asset, regardless of any caching policy set for the resource's old URL. :app:`Pyramid` can be configured to produce cache busting URLs for static -assets by passing the optional argument, ``cachebust`` to -:meth:`~pyramid.config.Configurator.add_static_view`: +assets using :meth:`~pyramid.config.Configurator.add_cache_buster`: .. code-block:: python :linenos: @@ -376,14 +375,18 @@ assets by passing the optional argument, ``cachebust`` to from pyramid.static import QueryStringConstantCacheBuster # config is an instance of pyramid.config.Configurator - config.add_static_view( - name='static', path='mypackage:folder/static', - cachebust=QueryStringConstantCacheBuster(str(int(time.time()))), - ) + config.add_static_view(name='static', path='mypackage:folder/static/') + config.add_cache_buster( + 'mypackage:folder/static/', + QueryStringConstantCacheBuster(str(int(time.time())))) + +.. note:: + The trailing slash on the ``add_cache_buster`` call is important to + indicate that it is overriding every asset in the folder and not just a + file named ``static``. -Setting the ``cachebust`` argument instructs :app:`Pyramid` to use a cache -busting scheme which adds the curent time for a static asset to the query -string in the asset's URL: +Adding the cachebuster instructs :app:`Pyramid` to add the current time for +a static asset to the query string in the asset's URL: .. code-block:: python :linenos: @@ -392,10 +395,7 @@ string in the asset's URL: # Returns: 'http://www.example.com/static/js/myapp.js?x=1445318121' When the web server restarts, the time constant will change and therefore so -will its URL. Supplying the ``cachebust`` argument also causes the static -view to set headers instructing clients to cache the asset for ten years, -unless the ``cache_max_age`` argument is also passed, in which case that -value is used. +will its URL. .. note:: @@ -410,7 +410,7 @@ Disabling the Cache Buster It can be useful in some situations (e.g., development) to globally disable all configured cache busters without changing calls to -:meth:`~pyramid.config.Configurator.add_static_view`. To do this set the +:meth:`~pyramid.config.Configurator.add_cache_buster`. To do this set the ``PYRAMID_PREVENT_CACHEBUST`` environment variable or the ``pyramid.prevent_cachebust`` configuration value to a true value. @@ -419,9 +419,9 @@ configured cache busters without changing calls to Customizing the Cache Buster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``cachebust`` option to -:meth:`~pyramid.config.Configurator.add_static_view` may be set to any object -that implements the interface :class:`~pyramid.interfaces.ICacheBuster`. +Calls to :meth:`~pyramid.config.Configurator.add_cache_buster` may use +any object that implements the interface +:class:`~pyramid.interfaces.ICacheBuster`. :app:`Pyramid` ships with a very simplistic :class:`~pyramid.static.QueryStringConstantCacheBuster`, which adds an @@ -461,8 +461,30 @@ the hash of the current commit: def tokenize(self, pathspec): return self.sha1 -Choosing a Cache Buster -~~~~~~~~~~~~~~~~~~~~~~~ +A simple cache buster that modifies the path segment can be constructed as +well: + +.. code-block:: python + :linenos: + + import posixpath + + def cache_buster(spec, subpath, kw): + base_subpath, ext = posixpath.splitext(subpath) + new_subpath = base_subpath + '-asdf' + ext + return new_subpath, kw + +The caveat with this approach is that modifying the path segment +changes the file name, and thus must match what is actually on the +filesystem in order for :meth:`~pyramid.config.Configurator.add_static_view` +to find the file. It's better to use the +:class:`~pyramid.static.ManifestCacheBuster` for these situations, as +described in the next section. + +.. _path_segment_cache_busters: + +Path Segments and Choosing a Cache Buster +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Many caching HTTP proxies will fail to cache a resource if the URL contains a query string. Therefore, in general, you should prefer a cache busting @@ -504,8 +526,11 @@ The following code would set up a cachebuster: config.add_static_view( name='http://mycdn.example.com/', - path='mypackage:static', - cachebust=ManifestCacheBuster('myapp:static/manifest.json')) + path='mypackage:static') + + config.add_cache_buster( + 'mypackage:static/', + ManifestCacheBuster('myapp:static/manifest.json')) A simpler approach is to use the :class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global @@ -524,8 +549,11 @@ start up as a cachebust token: config.add_static_view( name='http://mycdn.example.com/', - path='mypackage:static', - cachebust=QueryStringConstantCacheBuster(str(int(time.time())))) + path='mypackage:static') + + config.add_cache_buster( + 'mypackage:static/', + QueryStringConstantCacheBuster(str(int(time.time())))) .. index:: single: static assets view @@ -536,25 +564,38 @@ CSS and JavaScript source and cache busting Often one needs to refer to images and other static assets inside CSS and JavaScript files. If cache busting is active, the final static asset URL is not available until the static assets have been assembled. These URLs cannot be -handwritten. Thus, when having static asset references in CSS and JavaScript, -one needs to perform one of the following tasks: +handwritten. Below is an example of how to integrate the cache buster into +the entire stack. Remember, it is just an example and should be modified to +fit your specific tools. -* Process the files by using a precompiler which rewrites URLs to their final - cache busted form. Then, you can use the +* First, process the files by using a precompiler which rewrites URLs to their + final cache-busted form. Then, you can use the :class:`~pyramid.static.ManifestCacheBuster` to synchronize your asset pipeline with :app:`Pyramid`, allowing the pipeline to have full control over the final URLs of your assets. -* Templatize JS and CSS, and call ``request.static_url()`` inside their - template code. +Now that you are able to generate static URLs within :app:`Pyramid`, +you'll need to handle URLs that are out of our control. To do this you may +use some of the following options to get started: -* Pass static URL references to CSS and JavaScript via other means. +* Configure your asset pipeline to rewrite URL references inline in + CSS and JavaScript. This is the best approach because then the files + may be hosted by :app:`Pyramid` or an external CDN without haven't to + change anything. They really are static. + +* Templatize JS and CSS, and call ``request.static_url()`` inside their + template code. While this approach may work in certain scenarios, it is not + recommended because your static assets will not really be static and are now + dependent on :app:`Pyramid` to be served correctly. See + :ref:`advanced static` for more information on this approach. If your CSS and JavaScript assets use URLs to reference other assets it is recommended that you implement an external asset pipeline that can rewrite the generated static files with new URLs containing cache busting tokens. The machinery inside :app:`Pyramid` will not help with this step as it has very -little knowledge of the asset types your application may use. +little knowledge of the asset types your application may use. The integration +into :app:`Pyramid` is simply for linking those assets into your HTML and +other dynamic content. .. _advanced_static: -- cgit v1.2.3 From ffad12b0ac1ee24ad12d6d1a2f300da1ec004010 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 11:24:35 -0600 Subject: pass the raw asset spec into the cache buster --- pyramid/config/views.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 1115ccffc..44003127a 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -2119,7 +2119,7 @@ class StaticURLInfo(object): def _bust_asset_path(self, registry, spec, subpath, kw): pkg_name, pkg_subpath = spec.split(':') - absspec = rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) + rawspec = None overrides = registry.queryUtility(IPackageOverrides, name=pkg_name) if overrides is not None: resource_name = posixpath.join(pkg_subpath, subpath) @@ -2130,6 +2130,9 @@ class StaticURLInfo(object): rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) break + if rawspec is None: + rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) + for base_spec, cachebust in reversed(self.cache_busters): if ( base_spec == rawspec or @@ -2139,6 +2142,6 @@ class StaticURLInfo(object): else base_spec.endswith('/') ) and rawspec.startswith(base_spec) ): - subpath, kw = cachebust(absspec, subpath, kw) + subpath, kw = cachebust(rawspec, subpath, kw) break return subpath, kw -- cgit v1.2.3 From edad4ceca802324194000a98f3b07da7cedee546 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 15:56:30 -0600 Subject: tweak ManifestCacheBuster to allow overriding parse_manifest without touching the file loading logic --- pyramid/static.py | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/pyramid/static.py b/pyramid/static.py index cda98bea4..9559cd881 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -227,7 +227,7 @@ class ManifestCacheBuster(object): "images/background.png": "images/background-a8169106.png", } - Specifically, it is a JSON-serialized dictionary where the keys are the + By default, it is a JSON-serialized dictionary where the keys are the source asset paths used in calls to :meth:`~pyramid.request.Request.static_url`. For example:: @@ -236,6 +236,9 @@ class ManifestCacheBuster(object): >>> request.static_url('myapp:static/css/main.css') "http://www.example.com/static/css/main-678b7c80.css" + The file format and location can be changed by subclassing and overriding + :meth:`.parse_manifest`. + If a path is not found in the manifest it will pass through unchanged. If ``reload`` is ``True`` then the manifest file will be reloaded when @@ -244,11 +247,6 @@ class ManifestCacheBuster(object): If the manifest file cannot be found on disk it will be treated as an empty mapping unless ``reload`` is ``False``. - The default implementation assumes the requested (possibly cache-busted) - path is the actual filename on disk. Subclasses may override the ``match`` - method to alter this behavior. For example, to strip the cache busting - token from the path. - .. versionadded:: 1.6 """ exists = staticmethod(exists) # testing @@ -262,20 +260,23 @@ class ManifestCacheBuster(object): self._mtime = None if not reload: - self._manifest = self.parse_manifest() + self._manifest = self.get_manifest() - def parse_manifest(self): + def get_manifest(self): + with open(self.manifest_path, 'rb') as fp: + return self.parse_manifest(fp.read()) + + def parse_manifest(self, content): """ - Return a mapping parsed from the ``manifest_path``. + Parse the ``content`` read from the ``manifest_path`` into a + dictionary mapping. Subclasses may override this method to use something other than ``json.loads`` to load any type of file format and return a conforming dictionary. """ - with open(self.manifest_path, 'rb') as fp: - content = fp.read().decode('utf-8') - return json.loads(content) + return json.loads(content.decode('utf-8')) @property def manifest(self): @@ -285,7 +286,7 @@ class ManifestCacheBuster(object): return {} mtime = self.getmtime(self.manifest_path) if self._mtime is None or mtime > self._mtime: - self._manifest = self.parse_manifest() + self._manifest = self.get_manifest() self._mtime = mtime return self._manifest -- cgit v1.2.3 From b2fc4ace7fdb1dd2e90d6d3cc82f7b7b923ffa68 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 13:19:56 -0600 Subject: update docs on pathspec arg to ICacheBuster --- pyramid/interfaces.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/pyramid/interfaces.py b/pyramid/interfaces.py index bdf5bdfbe..153fdad03 100644 --- a/pyramid/interfaces.py +++ b/pyramid/interfaces.py @@ -1204,8 +1204,16 @@ class ICacheBuster(Interface): The ``kw`` argument is a dict of keywords that are to be passed eventually to :meth:`~pyramid.request.Request.static_url` for URL generation. The return value should be a two-tuple of - ``(subpath, kw)`` which are versions of the same arguments modified - to include the cache bust token in the generated URL. + ``(subpath, kw)`` where ``subpath`` is the relative URL from where the + file is served and ``kw`` is the same input argument. The return value + should be modified to include the cache bust token in the generated + URL. + + The ``pathspec`` refers to original location of the file, ignoring any + calls to :meth:`pyramid.config.Configurator.override_asset`. For + example, with a call ``request.static_url('myapp:static/foo.png'), the + ``pathspec`` may be ``themepkg:bar.png``, assuming a call to + ``config.override_asset('myapp:static/foo.png', 'themepkg:bar.png')``. """ # configuration phases: a lower phase number means the actions associated -- cgit v1.2.3 From 6923cae7f493c39b17367a3935a26065d4795ea6 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Tue, 8 Dec 2015 16:38:56 -0600 Subject: support cache busting only full folders --- pyramid/config/views.py | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 44003127a..ed7ae42ce 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1943,15 +1943,18 @@ class ViewsConfiguratorMixin(object): def add_cache_buster(self, path, cachebust): """ - The ``cachebust`` keyword argument may be set to cause + Add a cache buster to a set of files on disk. + + The ``path`` should be the path on disk where the static files + reside. This can be an absolute path, a package-relative path, or a + :term:`asset specification`. + + The ``cachebust`` argument may be set to cause :meth:`~pyramid.request.Request.static_url` to use cache busting when generating URLs. See :ref:`cache_busting` for general information about cache busting. The value of the ``cachebust`` argument must be an object which implements - :class:`~pyramid.interfaces.ICacheBuster`. If the ``cachebust`` - argument is provided, the default for ``cache_max_age`` is modified - to be ten years. ``cache_max_age`` may still be explicitly provided - to override this default. + :class:`~pyramid.interfaces.ICacheBuster`. """ spec = self._make_spec(path) @@ -2096,6 +2099,15 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) def add_cache_buster(self, config, spec, cachebust): + # ensure the spec always has a trailing slash as we only support + # adding cache busters to folders, not files + if os.path.isabs(spec): # FBO windows + sep = os.sep + else: + sep = '/' + if not spec.endswith(sep) and not spec.endswith(':'): + spec = spec + sep + def register(): cache_busters = self.cache_busters @@ -2134,14 +2146,7 @@ class StaticURLInfo(object): rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) for base_spec, cachebust in reversed(self.cache_busters): - if ( - base_spec == rawspec or - ( - base_spec.endswith(os.sep) - if os.path.isabs(base_spec) - else base_spec.endswith('/') - ) and rawspec.startswith(base_spec) - ): + if rawspec.startswith(base_spec): subpath, kw = cachebust(rawspec, subpath, kw) break return subpath, kw -- cgit v1.2.3 From 5e3439059daa94543f9437a280fed8d804cc7596 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 9 Dec 2015 20:34:54 -0600 Subject: fix broken tests --- pyramid/config/views.py | 35 ++++--- pyramid/tests/test_config/test_views.py | 173 ++++++++++++++++++++++---------- pyramid/tests/test_static.py | 74 +++++++------- 3 files changed, 178 insertions(+), 104 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index ed7ae42ce..e5bf1203e 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -38,6 +38,7 @@ from pyramid.interfaces import ( from pyramid import renderers +from pyramid.asset import resolve_asset_spec from pyramid.compat import ( string_types, urlparse, @@ -1985,14 +1986,12 @@ class StaticURLInfo(object): self.cache_busters = [] def generate(self, path, request, **kw): - disable_cache_buster = ( - request.registry.settings['pyramid.prevent_cachebust']) for (url, spec, route_name) in self.registrations: if path.startswith(spec): subpath = path[len(spec):] if WIN: # pragma: no cover subpath = subpath.replace('\\', '/') # windows - if not disable_cache_buster: + if self.cache_busters: subpath, kw = self._bust_asset_path( request.registry, spec, subpath, kw) if url is None: @@ -2099,6 +2098,9 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) def add_cache_buster(self, config, spec, cachebust): + if config.registry.settings.get('pyramid.prevent_cachebust'): + return + # ensure the spec always has a trailing slash as we only support # adding cache busters to folders, not files if os.path.isabs(spec): # FBO windows @@ -2130,20 +2132,25 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) def _bust_asset_path(self, registry, spec, subpath, kw): - pkg_name, pkg_subpath = spec.split(':') + pkg_name, pkg_subpath = resolve_asset_spec(spec) rawspec = None - overrides = registry.queryUtility(IPackageOverrides, name=pkg_name) - if overrides is not None: - resource_name = posixpath.join(pkg_subpath, subpath) - sources = overrides.filtered_sources(resource_name) - for source, filtered_path in sources: - rawspec = source.get_path(filtered_path) - if hasattr(source, 'pkg_name'): - rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) - break + + if pkg_name is not None: + overrides = registry.queryUtility(IPackageOverrides, name=pkg_name) + if overrides is not None: + resource_name = posixpath.join(pkg_subpath, subpath) + sources = overrides.filtered_sources(resource_name) + for source, filtered_path in sources: + rawspec = source.get_path(filtered_path) + if hasattr(source, 'pkg_name'): + rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) + break + + if rawspec is None: + rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) if rawspec is None: - rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) + rawspec = pkg_subpath + subpath for base_spec, cachebust in reversed(self.cache_busters): if rawspec.startswith(base_spec): diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index 020ed131d..eda8d8b05 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -1,3 +1,4 @@ +import os import unittest from pyramid import testing @@ -3887,49 +3888,35 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_registration_miss(self): inst = self._makeOne() - registrations = [ - (None, 'spec', 'route_name', None), - ('http://example.com/foo/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [ + (None, 'spec', 'route_name'), + ('http://example.com/foo/', 'package:path/', None)] request = self._makeRequest() result = inst.generate('package:path/abc', request) self.assertEqual(result, 'http://example.com/foo/abc') - def test_generate_registration_no_registry_on_request(self): - inst = self._makeOne() - registrations = [ - ('http://example.com/foo/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations - request = self._makeRequest() - del request.registry - result = inst.generate('package:path/abc', request) - self.assertEqual(result, 'http://example.com/foo/abc') - def test_generate_slash_in_name1(self): inst = self._makeOne() - registrations = [ - ('http://example.com/foo/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [('http://example.com/foo/', 'package:path/', None)] request = self._makeRequest() result = inst.generate('package:path/abc', request) self.assertEqual(result, 'http://example.com/foo/abc') def test_generate_slash_in_name2(self): inst = self._makeOne() - registrations = [ - ('http://example.com/foo/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [('http://example.com/foo/', 'package:path/', None)] request = self._makeRequest() result = inst.generate('package:path/', request) self.assertEqual(result, 'http://example.com/foo/') def test_generate_quoting(self): + from pyramid.interfaces import IStaticURLInfo config = testing.setUp() try: config.add_static_view('images', path='mypkg:templates') - inst = self._makeOne() request = testing.DummyRequest() request.registry = config.registry + inst = config.registry.getUtility(IStaticURLInfo) result = inst.generate('mypkg:templates/foo%2Fbar', request) self.assertEqual(result, 'http://example.com/images/foo%252Fbar') finally: @@ -3937,8 +3924,7 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_route_url(self): inst = self._makeOne() - registrations = [(None, 'package:path/', '__viewname/', None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [(None, 'package:path/', '__viewname/')] def route_url(n, **kw): self.assertEqual(n, '__viewname/') self.assertEqual(kw, {'subpath':'abc', 'a':1}) @@ -3950,8 +3936,7 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_url_unquoted_local(self): inst = self._makeOne() - registrations = [(None, 'package:path/', '__viewname/', None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [(None, 'package:path/', '__viewname/')] def route_url(n, **kw): self.assertEqual(n, '__viewname/') self.assertEqual(kw, {'subpath':'abc def', 'a':1}) @@ -3963,16 +3948,15 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_url_quoted_remote(self): inst = self._makeOne() - registrations = [('http://example.com/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [('http://example.com/', 'package:path/', None)] request = self._makeRequest() result = inst.generate('package:path/abc def', request, a=1) self.assertEqual(result, 'http://example.com/abc%20def') def test_generate_url_with_custom_query(self): inst = self._makeOne() - registrations = [('http://example.com/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + registrations = [('http://example.com/', 'package:path/', None)] + inst.registrations = registrations request = self._makeRequest() result = inst.generate('package:path/abc def', request, a=1, _query='(openlayers)') @@ -3981,12 +3965,10 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_url_with_custom_anchor(self): inst = self._makeOne() - registrations = [('http://example.com/', 'package:path/', None, None)] - inst._get_registrations = lambda *x: registrations + inst.registrations = [('http://example.com/', 'package:path/', None)] request = self._makeRequest() uc = text_(b'La Pe\xc3\xb1a', 'utf-8') - result = inst.generate('package:path/abc def', request, a=1, - _anchor=uc) + result = inst.generate('package:path/abc def', request, a=1, _anchor=uc) self.assertEqual(result, 'http://example.com/abc%20def#La%20Pe%C3%B1a') @@ -3995,52 +3977,102 @@ class TestStaticURLInfo(unittest.TestCase): kw['foo'] = 'bar' return 'foo' + '/' + subpath, kw inst = self._makeOne() - inst.registrations = [(None, 'package:path/', '__viewname', cachebust)] + inst.registrations = [(None, 'package:path/', '__viewname')] inst.cache_busters = [('package:path/', cachebust)] request = self._makeRequest() + called = [False] def route_url(n, **kw): + called[0] = True self.assertEqual(n, '__viewname') - self.assertEqual(kw, {'subpath':'foo/abc', 'foo':'bar'}) + self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar'}) request.route_url = route_url inst.generate('package:path/abc', request) + self.assertTrue(called[0]) + + def test_generate_url_cachebust_abspath(self): + here = os.path.dirname(__file__) + os.sep + def cachebust(pathspec, subpath, kw): + kw['foo'] = 'bar' + return 'foo' + '/' + subpath, kw + inst = self._makeOne() + inst.registrations = [(None, here, '__viewname')] + inst.cache_busters = [(here, cachebust)] + request = self._makeRequest() + called = [False] + def route_url(n, **kw): + called[0] = True + self.assertEqual(n, '__viewname') + self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar'}) + request.route_url = route_url + inst.generate(here + 'abc', request) + self.assertTrue(called[0]) + + def test_generate_url_cachebust_nomatch(self): + def fake_cb(*a, **kw): raise AssertionError + inst = self._makeOne() + inst.registrations = [(None, 'package:path/', '__viewname')] + inst.cache_busters = [('package:path2/', fake_cb)] + request = self._makeRequest() + called = [False] + def route_url(n, **kw): + called[0] = True + self.assertEqual(n, '__viewname') + self.assertEqual(kw, {'subpath': 'abc'}) + request.route_url = route_url + inst.generate('package:path/abc', request) + self.assertTrue(called[0]) + + def test_generate_url_cachebust_with_overrides(self): + config = testing.setUp() + try: + config.add_static_view('static', 'path') + config.override_asset( + 'pyramid.tests.test_config:path/', + 'pyramid.tests.test_config:other_path/') + def cb(pathspec, subpath, kw): + kw['_query'] = {'x': 'foo'} + return subpath, kw + config.add_cache_buster('other_path', cb) + request = testing.DummyRequest() + result = request.static_url('path/foo.png') + self.assertEqual(result, 'http://example.com/static/foo.png?x=foo') + finally: + testing.tearDown() def test_add_already_exists(self): config = DummyConfig() inst = self._makeOne() inst.registrations = [('http://example.com/', 'package:path/', None)] inst.add(config, 'http://example.com', 'anotherpackage:path') - expected = [ - ('http://example.com/', 'anotherpackage:path/', None, None)] + expected = [('http://example.com/', 'anotherpackage:path/', None)] self.assertEqual(inst.registrations, expected) def test_add_package_root(self): config = DummyConfig() inst = self._makeOne() inst.add(config, 'http://example.com', 'package:') - expected = [('http://example.com/', 'package:', None, None)] + expected = [('http://example.com/', 'package:', None)] self.assertEqual(inst.registrations, expected) def test_add_url_withendslash(self): config = DummyConfig() inst = self._makeOne() inst.add(config, 'http://example.com/', 'anotherpackage:path') - expected = [ - ('http://example.com/', 'anotherpackage:path/', None, None)] + expected = [('http://example.com/', 'anotherpackage:path/', None)] self.assertEqual(inst.registrations, expected) def test_add_url_noendslash(self): config = DummyConfig() inst = self._makeOne() inst.add(config, 'http://example.com', 'anotherpackage:path') - expected = [ - ('http://example.com/', 'anotherpackage:path/', None, None)] + expected = [('http://example.com/', 'anotherpackage:path/', None)] self.assertEqual(inst.registrations, expected) def test_add_url_noscheme(self): config = DummyConfig() inst = self._makeOne() inst.add(config, '//example.com', 'anotherpackage:path') - expected = [('//example.com/', 'anotherpackage:path/', None, None)] + expected = [('//example.com/', 'anotherpackage:path/', None)] self.assertEqual(inst.registrations, expected) def test_add_viewname(self): @@ -4049,7 +4081,7 @@ class TestStaticURLInfo(unittest.TestCase): config = DummyConfig() inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path', cache_max_age=1) - expected = [(None, 'anotherpackage:path/', '__view/', None)] + expected = [(None, 'anotherpackage:path/', '__view/')] self.assertEqual(inst.registrations, expected) self.assertEqual(config.route_args, ('__view/', 'view/*subpath')) self.assertEqual(config.view_kw['permission'], NO_PERMISSION_REQUIRED) @@ -4060,7 +4092,7 @@ class TestStaticURLInfo(unittest.TestCase): config.route_prefix = '/abc' inst = self._makeOne() inst.add(config, 'view', 'anotherpackage:path',) - expected = [(None, 'anotherpackage:path/', '__/abc/view/', None)] + expected = [(None, 'anotherpackage:path/', '__/abc/view/')] self.assertEqual(inst.registrations, expected) self.assertEqual(config.route_args, ('__/abc/view/', 'view/*subpath')) @@ -4097,20 +4129,47 @@ class TestStaticURLInfo(unittest.TestCase): config = DummyConfig() config.registry.settings['pyramid.prevent_cachebust'] = True inst = self._makeOne() - inst.add(config, 'view', 'mypackage:path', cachebust=True) - cachebust = config.registry._static_url_registrations[0][3] - self.assertEqual(cachebust, None) + cachebust = DummyCacheBuster('foo') + inst.add_cache_buster(config, 'mypackage:path', cachebust) + self.assertEqual(inst.cache_busters, []) - def test_add_cachebust_custom(self): + def test_add_cachebuster(self): config = DummyConfig() inst = self._makeOne() - inst.add(config, 'view', 'mypackage:path', - cachebust=DummyCacheBuster('foo')) - cachebust = config.registry._static_url_registrations[0][3] - subpath, kw = cachebust('some/path', {}) + inst.add_cache_buster(config, 'mypackage:path', DummyCacheBuster('foo')) + cachebust = inst.cache_busters[-1][1] + subpath, kw = cachebust('mypackage:some/path', 'some/path', {}) self.assertEqual(subpath, 'some/path') self.assertEqual(kw['x'], 'foo') + def test_add_cachebuster_abspath(self): + here = os.path.dirname(__file__) + config = DummyConfig() + inst = self._makeOne() + cb = DummyCacheBuster('foo') + inst.add_cache_buster(config, here, cb) + self.assertEqual(inst.cache_busters, [(here + '/', cb)]) + + def test_add_cachebuster_overwrite(self): + config = DummyConfig() + inst = self._makeOne() + cb1 = DummyCacheBuster('foo') + cb2 = DummyCacheBuster('bar') + inst.add_cache_buster(config, 'mypackage:path/', cb1) + inst.add_cache_buster(config, 'mypackage:path', cb2) + self.assertEqual(inst.cache_busters, + [('mypackage:path/', cb2)]) + + def test_add_cachebuster_for_more_specific_path(self): + config = DummyConfig() + inst = self._makeOne() + cb1 = DummyCacheBuster('foo') + cb2 = DummyCacheBuster('bar') + inst.add_cache_buster(config, 'mypackage:path', cb1) + inst.add_cache_buster(config, 'mypackage:path/sub', cb2) + self.assertEqual(inst.cache_busters, + [('mypackage:path/', cb1), ('mypackage:path/sub/', cb2)]) + class Test_view_description(unittest.TestCase): def _callFUT(self, view): from pyramid.config.views import view_description @@ -4130,9 +4189,14 @@ class Test_view_description(unittest.TestCase): class DummyRegistry: + utility = None + def __init__(self): self.settings = {} + def queryUtility(self, type_or_iface, name=None, default=None): + return self.utility or default + from zope.interface import implementer from pyramid.interfaces import ( IResponse, @@ -4193,6 +4257,9 @@ class DummySecurityPolicy: return self.permitted class DummyConfig: + def __init__(self): + self.registry = DummyRegistry() + route_prefix = '' def add_route(self, *args, **kw): self.route_args = args diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index 4a07c2cb1..73f242add 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -385,29 +385,29 @@ class TestQueryStringConstantCacheBuster(unittest.TestCase): fut = self._makeOne().tokenize self.assertEqual(fut('whatever'), 'foo') - def test_pregenerate(self): - fut = self._makeOne().pregenerate + def test_it(self): + fut = self._makeOne() self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'x': 'foo'}})) + fut('foo', 'bar', {}), + ('bar', {'_query': {'x': 'foo'}})) - def test_pregenerate_change_param(self): - fut = self._makeOne('y').pregenerate + def test_change_param(self): + fut = self._makeOne('y') self.assertEqual( - fut('foo', ('bar',), {}), - (('bar',), {'_query': {'y': 'foo'}})) + fut('foo', 'bar', {}), + ('bar', {'_query': {'y': 'foo'}})) - def test_pregenerate_query_is_already_tuples(self): - fut = self._makeOne().pregenerate + def test_query_is_already_tuples(self): + fut = self._makeOne() self.assertEqual( - fut('foo', ('bar',), {'_query': [('a', 'b')]}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) + fut('foo', 'bar', {'_query': [('a', 'b')]}), + ('bar', {'_query': (('a', 'b'), ('x', 'foo'))})) - def test_pregenerate_query_is_tuple_of_tuples(self): - fut = self._makeOne().pregenerate + def test_query_is_tuple_of_tuples(self): + fut = self._makeOne() self.assertEqual( - fut('foo', ('bar',), {'_query': (('a', 'b'),)}), - (('bar',), {'_query': (('a', 'b'), ('x', 'foo'))})) + fut('foo', 'bar', {'_query': (('a', 'b'),)}), + ('bar', {'_query': (('a', 'b'), ('x', 'foo'))})) class TestManifestCacheBuster(unittest.TestCase): @@ -417,55 +417,55 @@ class TestManifestCacheBuster(unittest.TestCase): def test_it(self): manifest_path = os.path.join(here, 'fixtures', 'manifest.json') - fut = self._makeOne(manifest_path).pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + fut = self._makeOne(manifest_path) + self.assertEqual(fut('foo', 'bar', {}), ('bar', {})) self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-test.css', {})) def test_it_with_relspec(self): - fut = self._makeOne('fixtures/manifest.json').pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + fut = self._makeOne('fixtures/manifest.json') + self.assertEqual(fut('foo', 'bar', {}), ('bar', {})) self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-test.css', {})) def test_it_with_absspec(self): - fut = self._makeOne('pyramid.tests:fixtures/manifest.json').pregenerate - self.assertEqual(fut('foo', ('bar',), {}), (['bar'], {})) + fut = self._makeOne('pyramid.tests:fixtures/manifest.json') + self.assertEqual(fut('foo', 'bar', {}), ('bar', {})) self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-test.css', {})) def test_reload(self): manifest_path = os.path.join(here, 'fixtures', 'manifest.json') new_manifest_path = os.path.join(here, 'fixtures', 'manifest2.json') inst = self._makeOne('foo', reload=True) inst.getmtime = lambda *args, **kwargs: 0 - fut = inst.pregenerate + fut = inst # test without a valid manifest self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main.css', {})) # swap to a real manifest, setting mtime to 0 inst.manifest_path = manifest_path self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-test.css', {})) # ensure switching the path doesn't change the result inst.manifest_path = new_manifest_path self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-test.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-test.css', {})) # update mtime, should cause a reload inst.getmtime = lambda *args, **kwargs: 1 self.assertEqual( - fut('foo', ('css', 'main.css'), {}), - (['css', 'main-678b7c80.css'], {})) + fut('foo', 'css/main.css', {}), + ('css/main-678b7c80.css', {})) def test_invalid_manifest(self): self.assertRaises(IOError, lambda: self._makeOne('foo')) -- cgit v1.2.3 From 3175c990bc02805b729594996f6360b81f5f0ebc Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 9 Dec 2015 23:01:35 -0600 Subject: fix broken ref in docs --- docs/narr/assets.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 8b41f9b8a..0e3f6af11 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -587,7 +587,7 @@ use some of the following options to get started: template code. While this approach may work in certain scenarios, it is not recommended because your static assets will not really be static and are now dependent on :app:`Pyramid` to be served correctly. See - :ref:`advanced static` for more information on this approach. + :ref:`advanced_static` for more information on this approach. If your CSS and JavaScript assets use URLs to reference other assets it is recommended that you implement an external asset pipeline that can rewrite the -- cgit v1.2.3 From 4350699a208dc9304ae8c8dd165251f227ff5189 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 9 Dec 2015 23:02:28 -0600 Subject: remove unused import --- pyramid/config/views.py | 1 - 1 file changed, 1 deletion(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index e5bf1203e..1fcdcb136 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -69,7 +69,6 @@ from pyramid.response import Response from pyramid.security import NO_PERMISSION_REQUIRED from pyramid.static import static_view -from pyramid.threadlocal import get_current_registry from pyramid.url import parse_url_overrides -- cgit v1.2.3 From fb9b9f106699b38bc49c14c751d1b948a3c08533 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 11 Dec 2015 03:33:37 -0800 Subject: add developing and contributing section - wrap to 79 columns --- README.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index e99133441..fed4534a0 100644 --- a/README.rst +++ b/README.rst @@ -16,9 +16,9 @@ Pyramid :target: https://webchat.freenode.net/?channels=pyramid :alt: IRC Freenode -Pyramid is a small, fast, down-to-earth, open source Python web framework. -It makes real-world web application development and -deployment more fun, more predictable, and more productive. +Pyramid is a small, fast, down-to-earth, open source Python web framework. It +makes real-world web application development and deployment more fun, more +predictable, and more productive. Pyramid is produced by the `Pylons Project `_. @@ -28,6 +28,13 @@ Support and Documentation See the `Pylons Project website `_ to view documentation, report bugs, and obtain support. +Developing and Contributing +--------------------------- + +See `HACKING.txt` and `contributing.md` for guidelines for running tests, +adding features, coding style, and updating documentation when developing in or +contributing to Pyramid. + License ------- -- cgit v1.2.3 From 4bd36e8669fece803ff8bb209e655ab16fb8c63b Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Fri, 11 Dec 2015 03:44:01 -0800 Subject: use double backticks for inline code formatting of filename --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index fed4534a0..2237d9950 100644 --- a/README.rst +++ b/README.rst @@ -31,7 +31,7 @@ documentation, report bugs, and obtain support. Developing and Contributing --------------------------- -See `HACKING.txt` and `contributing.md` for guidelines for running tests, +See ``HACKING.txt`` and ``contributing.md`` for guidelines for running tests, adding features, coding style, and updating documentation when developing in or contributing to Pyramid. -- cgit v1.2.3 From 3241405cefdb2cc545d689140728a5d22be7d18a Mon Sep 17 00:00:00 2001 From: Sri Sanketh Uppalapati Date: Sat, 12 Dec 2015 16:44:54 +0530 Subject: Update url.py --- pyramid/url.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/pyramid/url.py b/pyramid/url.py index b004c40ec..210163fa5 100644 --- a/pyramid/url.py +++ b/pyramid/url.py @@ -96,19 +96,17 @@ class URLMethodsMixin(object): if scheme == 'http': if port is None: port = '80' - url = scheme + '://' - if port is not None: - port = str(port) if host is None: host = e.get('HTTP_HOST') - if host is None: - host = e['SERVER_NAME'] + if host is None: + host = e['SERVER_NAME'] if port is None: if ':' in host: host, port = host.split(':', 1) else: port = e['SERVER_PORT'] else: + port=str(port) if ':' in host: host, _ = host.split(':', 1) if scheme == 'https': @@ -117,7 +115,7 @@ class URLMethodsMixin(object): elif scheme == 'http': if port == '80': port = None - url += host + url = scheme + '://' + host if port: url += ':%s' % port -- cgit v1.2.3 From 55983f327b39bbf265b1ace1baac5a1f58c0f2fe Mon Sep 17 00:00:00 2001 From: Sri Sanketh Uppalapati Date: Sat, 12 Dec 2015 16:55:17 +0530 Subject: Update url.py --- pyramid/url.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyramid/url.py b/pyramid/url.py index 210163fa5..fd62f0057 100644 --- a/pyramid/url.py +++ b/pyramid/url.py @@ -106,7 +106,7 @@ class URLMethodsMixin(object): else: port = e['SERVER_PORT'] else: - port=str(port) + port = str(port) if ':' in host: host, _ = host.split(':', 1) if scheme == 'https': -- cgit v1.2.3 From 7682837de754c2515ff617a87afecae2498cfa6e Mon Sep 17 00:00:00 2001 From: Sri Sanketh Uppalapati Date: Sat, 12 Dec 2015 17:02:31 +0530 Subject: Update url.py --- pyramid/url.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/pyramid/url.py b/pyramid/url.py index fd62f0057..812514638 100644 --- a/pyramid/url.py +++ b/pyramid/url.py @@ -98,8 +98,6 @@ class URLMethodsMixin(object): port = '80' if host is None: host = e.get('HTTP_HOST') - if host is None: - host = e['SERVER_NAME'] if port is None: if ':' in host: host, port = host.split(':', 1) -- cgit v1.2.3 From b4fcff0471e16b6dab3a685df970bca712c5cb3b Mon Sep 17 00:00:00 2001 From: Sri Sanketh Uppalapati Date: Sat, 12 Dec 2015 17:06:29 +0530 Subject: Update url.py --- pyramid/url.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pyramid/url.py b/pyramid/url.py index 812514638..fd62f0057 100644 --- a/pyramid/url.py +++ b/pyramid/url.py @@ -98,6 +98,8 @@ class URLMethodsMixin(object): port = '80' if host is None: host = e.get('HTTP_HOST') + if host is None: + host = e['SERVER_NAME'] if port is None: if ':' in host: host, port = host.split(':', 1) -- cgit v1.2.3 From 3b75ea56485ee2b5f10f757fce31c00ca54dbc86 Mon Sep 17 00:00:00 2001 From: Sri Sanketh Uppalapati Date: Sat, 12 Dec 2015 17:23:43 +0530 Subject: Update CONTRIBUTORS.txt --- CONTRIBUTORS.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt index 4edf1b4e9..1f3597e84 100644 --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -256,3 +256,5 @@ Contributors - Amos Latteier, 2015/10/22 - Rami Chousein, 2015/10/28 + +- Sri Sanketh Uppalapati, 2015/12/12 -- cgit v1.2.3 From 4d19b84cb8134a0e7f030064e5d944defaa6970a Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Mon, 14 Dec 2015 00:17:39 -0600 Subject: new default behavior matches virtual specs, old behavior hidden behind explicit=True --- pyramid/config/views.py | 67 ++++++++++++++++++++++---------- pyramid/interfaces.py | 26 +++++++++---- pyramid/static.py | 10 ++--- pyramid/tests/test_config/test_views.py | 69 ++++++++++++++++++++++++--------- pyramid/tests/test_static.py | 2 +- 5 files changed, 123 insertions(+), 51 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 1fcdcb136..759276351 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -1,4 +1,3 @@ -import bisect import inspect import posixpath import operator @@ -1941,7 +1940,7 @@ class ViewsConfiguratorMixin(object): info = self._get_static_info() info.add(self, name, spec, **kw) - def add_cache_buster(self, path, cachebust): + def add_cache_buster(self, path, cachebust, explicit=False): """ Add a cache buster to a set of files on disk. @@ -1956,10 +1955,16 @@ class ViewsConfiguratorMixin(object): be an object which implements :class:`~pyramid.interfaces.ICacheBuster`. + If ``explicit`` is set to ``True`` then the ``path`` for the cache + buster will be matched based on the ``rawspec`` instead of the + ``pathspec`` as defined in the + :class:`~pyramid.interfaces.ICacheBuster` interface. + Default: ``False``. + """ spec = self._make_spec(path) info = self._get_static_info() - info.add_cache_buster(self, spec, cachebust) + info.add_cache_buster(self, spec, cachebust, explicit=explicit) def _get_static_info(self): info = self.registry.queryUtility(IStaticURLInfo) @@ -1992,7 +1997,7 @@ class StaticURLInfo(object): subpath = subpath.replace('\\', '/') # windows if self.cache_busters: subpath, kw = self._bust_asset_path( - request.registry, spec, subpath, kw) + request, spec, subpath, kw) if url is None: kw['subpath'] = subpath return request.route_url(route_name, **kw) @@ -2096,7 +2101,7 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) - def add_cache_buster(self, config, spec, cachebust): + def add_cache_buster(self, config, spec, cachebust, explicit=False): if config.registry.settings.get('pyramid.prevent_cachebust'): return @@ -2112,29 +2117,46 @@ class StaticURLInfo(object): def register(): cache_busters = self.cache_busters - specs = [t[0] for t in cache_busters] - if spec in specs: - idx = specs.index(spec) - cache_busters.pop(idx) + # find duplicate cache buster (old_idx) + # and insertion location (new_idx) + new_idx, old_idx = len(cache_busters), None + for idx, (spec_, cb_, explicit_) in enumerate(cache_busters): + # if we find an identical (spec, explicit) then use it + if spec == spec_ and explicit == explicit_: + old_idx = new_idx = idx + break + + # past all explicit==False specs then add to the end + elif not explicit and explicit_: + new_idx = idx + break + + # explicit matches and spec is shorter + elif explicit == explicit_ and len(spec) < len(spec_): + new_idx = idx + break - lengths = [len(t[0]) for t in cache_busters] - new_idx = bisect.bisect_left(lengths, len(spec)) - cache_busters.insert(new_idx, (spec, cachebust)) + if old_idx is not None: + cache_busters.pop(old_idx) + cache_busters.insert(new_idx, (spec, cachebust, explicit)) intr = config.introspectable('cache busters', spec, 'cache buster for %r' % spec, 'cache buster') intr['cachebust'] = cachebust - intr['spec'] = spec + intr['path'] = spec + intr['explicit'] = explicit config.action(None, callable=register, introspectables=(intr,)) - def _bust_asset_path(self, registry, spec, subpath, kw): + def _bust_asset_path(self, request, spec, subpath, kw): + registry = request.registry pkg_name, pkg_subpath = resolve_asset_spec(spec) rawspec = None if pkg_name is not None: + pathspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) overrides = registry.queryUtility(IPackageOverrides, name=pkg_name) if overrides is not None: resource_name = posixpath.join(pkg_subpath, subpath) @@ -2145,14 +2167,19 @@ class StaticURLInfo(object): rawspec = '{0}:{1}'.format(source.pkg_name, rawspec) break - if rawspec is None: - rawspec = '{0}:{1}{2}'.format(pkg_name, pkg_subpath, subpath) + else: + pathspec = pkg_subpath + subpath if rawspec is None: - rawspec = pkg_subpath + subpath + rawspec = pathspec - for base_spec, cachebust in reversed(self.cache_busters): - if rawspec.startswith(base_spec): - subpath, kw = cachebust(rawspec, subpath, kw) + kw['pathspec'] = pathspec + kw['rawspec'] = rawspec + for spec_, cachebust, explicit in reversed(self.cache_busters): + if ( + (explicit and rawspec.startswith(spec_)) or + (not explicit and pathspec.startswith(spec_)) + ): + subpath, kw = cachebust(request, subpath, kw) break return subpath, kw diff --git a/pyramid/interfaces.py b/pyramid/interfaces.py index 153fdad03..bbdc5121d 100644 --- a/pyramid/interfaces.py +++ b/pyramid/interfaces.py @@ -1194,11 +1194,11 @@ class ICacheBuster(Interface): .. versionadded:: 1.6 """ - def __call__(pathspec, subpath, kw): + def __call__(request, subpath, kw): """ Modifies a subpath and/or keyword arguments from which a static asset - URL will be computed during URL generation. The ``pathspec`` argument - is the path specification for the resource to be cache busted. + URL will be computed during URL generation. + The ``subpath`` argument is a path of ``/``-delimited segments that represent the portion of the asset URL which is used to find the asset. The ``kw`` argument is a dict of keywords that are to be passed @@ -1209,10 +1209,22 @@ class ICacheBuster(Interface): should be modified to include the cache bust token in the generated URL. - The ``pathspec`` refers to original location of the file, ignoring any - calls to :meth:`pyramid.config.Configurator.override_asset`. For - example, with a call ``request.static_url('myapp:static/foo.png'), the - ``pathspec`` may be ``themepkg:bar.png``, assuming a call to + The ``kw`` dictionary contains extra arguments passed to + :meth:`~pyramid.request.Request.static_url` as well as some extra + items that may be usful including: + + - ``pathspec`` is the path specification for the resource + to be cache busted. + + - ``rawspec`` is the original location of the file, ignoring + any calls to :meth:`pyramid.config.Configurator.override_asset`. + + The ``pathspec`` and ``rawspec`` values are only different in cases + where an asset has been mounted into a virtual location using + :meth:`pyramid.config.Configurator.override_asset`. For example, with + a call to ``request.static_url('myapp:static/foo.png'), the + ``pathspec`` is ``myapp:static/foo.png`` whereas the ``rawspec`` may + be ``themepkg:bar.png``, assuming a call to ``config.override_asset('myapp:static/foo.png', 'themepkg:bar.png')``. """ diff --git a/pyramid/static.py b/pyramid/static.py index 9559cd881..4054d5be0 100644 --- a/pyramid/static.py +++ b/pyramid/static.py @@ -172,15 +172,15 @@ class QueryStringCacheBuster(object): to the query string and defaults to ``'x'``. To use this class, subclass it and provide a ``tokenize`` method which - accepts a ``pathspec`` and returns a token. + accepts ``request, pathspec, kw`` and returns a token. .. versionadded:: 1.6 """ def __init__(self, param='x'): self.param = param - def __call__(self, pathspec, subpath, kw): - token = self.tokenize(pathspec) + def __call__(self, request, subpath, kw): + token = self.tokenize(request, subpath, kw) query = kw.setdefault('_query', {}) if isinstance(query, dict): query[self.param] = token @@ -205,7 +205,7 @@ class QueryStringConstantCacheBuster(QueryStringCacheBuster): super(QueryStringConstantCacheBuster, self).__init__(param=param) self._token = token - def tokenize(self, pathspec): + def tokenize(self, request, subpath, kw): return self._token class ManifestCacheBuster(object): @@ -290,6 +290,6 @@ class ManifestCacheBuster(object): self._mtime = mtime return self._manifest - def __call__(self, pathspec, subpath, kw): + def __call__(self, request, subpath, kw): subpath = self.manifest.get(subpath, subpath) return (subpath, kw) diff --git a/pyramid/tests/test_config/test_views.py b/pyramid/tests/test_config/test_views.py index eda8d8b05..e89d43c9a 100644 --- a/pyramid/tests/test_config/test_views.py +++ b/pyramid/tests/test_config/test_views.py @@ -3978,13 +3978,15 @@ class TestStaticURLInfo(unittest.TestCase): return 'foo' + '/' + subpath, kw inst = self._makeOne() inst.registrations = [(None, 'package:path/', '__viewname')] - inst.cache_busters = [('package:path/', cachebust)] + inst.cache_busters = [('package:path/', cachebust, False)] request = self._makeRequest() called = [False] def route_url(n, **kw): called[0] = True self.assertEqual(n, '__viewname') - self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar'}) + self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar', + 'pathspec': 'package:path/abc', + 'rawspec': 'package:path/abc'}) request.route_url = route_url inst.generate('package:path/abc', request) self.assertTrue(called[0]) @@ -3996,13 +3998,15 @@ class TestStaticURLInfo(unittest.TestCase): return 'foo' + '/' + subpath, kw inst = self._makeOne() inst.registrations = [(None, here, '__viewname')] - inst.cache_busters = [(here, cachebust)] + inst.cache_busters = [(here, cachebust, False)] request = self._makeRequest() called = [False] def route_url(n, **kw): called[0] = True self.assertEqual(n, '__viewname') - self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar'}) + self.assertEqual(kw, {'subpath': 'foo/abc', 'foo': 'bar', + 'pathspec': here + 'abc', + 'rawspec': here + 'abc'}) request.route_url = route_url inst.generate(here + 'abc', request) self.assertTrue(called[0]) @@ -4011,13 +4015,15 @@ class TestStaticURLInfo(unittest.TestCase): def fake_cb(*a, **kw): raise AssertionError inst = self._makeOne() inst.registrations = [(None, 'package:path/', '__viewname')] - inst.cache_busters = [('package:path2/', fake_cb)] + inst.cache_busters = [('package:path2/', fake_cb, False)] request = self._makeRequest() called = [False] def route_url(n, **kw): called[0] = True self.assertEqual(n, '__viewname') - self.assertEqual(kw, {'subpath': 'abc'}) + self.assertEqual(kw, {'subpath': 'abc', + 'pathspec': 'package:path/abc', + 'rawspec': 'package:path/abc'}) request.route_url = route_url inst.generate('package:path/abc', request) self.assertTrue(called[0]) @@ -4025,17 +4031,22 @@ class TestStaticURLInfo(unittest.TestCase): def test_generate_url_cachebust_with_overrides(self): config = testing.setUp() try: + request = testing.DummyRequest() config.add_static_view('static', 'path') config.override_asset( 'pyramid.tests.test_config:path/', 'pyramid.tests.test_config:other_path/') - def cb(pathspec, subpath, kw): - kw['_query'] = {'x': 'foo'} - return subpath, kw - config.add_cache_buster('other_path', cb) - request = testing.DummyRequest() + def cb(val): + def cb_(request, subpath, kw): + kw['_query'] = {'x': val} + return subpath, kw + return cb_ + config.add_cache_buster('path', cb('foo')) result = request.static_url('path/foo.png') self.assertEqual(result, 'http://example.com/static/foo.png?x=foo') + config.add_cache_buster('other_path', cb('bar'), explicit=True) + result = request.static_url('path/foo.png') + self.assertEqual(result, 'http://example.com/static/foo.png?x=bar') finally: testing.tearDown() @@ -4138,7 +4149,7 @@ class TestStaticURLInfo(unittest.TestCase): inst = self._makeOne() inst.add_cache_buster(config, 'mypackage:path', DummyCacheBuster('foo')) cachebust = inst.cache_busters[-1][1] - subpath, kw = cachebust('mypackage:some/path', 'some/path', {}) + subpath, kw = cachebust(None, 'some/path', {}) self.assertEqual(subpath, 'some/path') self.assertEqual(kw['x'], 'foo') @@ -4148,7 +4159,7 @@ class TestStaticURLInfo(unittest.TestCase): inst = self._makeOne() cb = DummyCacheBuster('foo') inst.add_cache_buster(config, here, cb) - self.assertEqual(inst.cache_busters, [(here + '/', cb)]) + self.assertEqual(inst.cache_busters, [(here + '/', cb, False)]) def test_add_cachebuster_overwrite(self): config = DummyConfig() @@ -4158,17 +4169,39 @@ class TestStaticURLInfo(unittest.TestCase): inst.add_cache_buster(config, 'mypackage:path/', cb1) inst.add_cache_buster(config, 'mypackage:path', cb2) self.assertEqual(inst.cache_busters, - [('mypackage:path/', cb2)]) + [('mypackage:path/', cb2, False)]) + + def test_add_cachebuster_overwrite_explicit(self): + config = DummyConfig() + inst = self._makeOne() + cb1 = DummyCacheBuster('foo') + cb2 = DummyCacheBuster('bar') + inst.add_cache_buster(config, 'mypackage:path/', cb1) + inst.add_cache_buster(config, 'mypackage:path', cb2, True) + self.assertEqual(inst.cache_busters, + [('mypackage:path/', cb1, False), + ('mypackage:path/', cb2, True)]) def test_add_cachebuster_for_more_specific_path(self): config = DummyConfig() inst = self._makeOne() cb1 = DummyCacheBuster('foo') cb2 = DummyCacheBuster('bar') + cb3 = DummyCacheBuster('baz') + cb4 = DummyCacheBuster('xyz') + cb5 = DummyCacheBuster('w') inst.add_cache_buster(config, 'mypackage:path', cb1) - inst.add_cache_buster(config, 'mypackage:path/sub', cb2) - self.assertEqual(inst.cache_busters, - [('mypackage:path/', cb1), ('mypackage:path/sub/', cb2)]) + inst.add_cache_buster(config, 'mypackage:path/sub', cb2, True) + inst.add_cache_buster(config, 'mypackage:path/sub/other', cb3) + inst.add_cache_buster(config, 'mypackage:path/sub/other', cb4, True) + inst.add_cache_buster(config, 'mypackage:path/sub/less', cb5, True) + self.assertEqual( + inst.cache_busters, + [('mypackage:path/', cb1, False), + ('mypackage:path/sub/other/', cb3, False), + ('mypackage:path/sub/', cb2, True), + ('mypackage:path/sub/less/', cb5, True), + ('mypackage:path/sub/other/', cb4, True)]) class Test_view_description(unittest.TestCase): def _callFUT(self, view): @@ -4293,7 +4326,7 @@ class DummyCacheBuster(object): def __init__(self, token): self.token = token - def __call__(self, pathspec, subpath, kw): + def __call__(self, request, subpath, kw): kw['x'] = self.token return subpath, kw diff --git a/pyramid/tests/test_static.py b/pyramid/tests/test_static.py index 73f242add..2ca86bc44 100644 --- a/pyramid/tests/test_static.py +++ b/pyramid/tests/test_static.py @@ -383,7 +383,7 @@ class TestQueryStringConstantCacheBuster(unittest.TestCase): def test_token(self): fut = self._makeOne().tokenize - self.assertEqual(fut('whatever'), 'foo') + self.assertEqual(fut(None, 'whatever', None), 'foo') def test_it(self): fut = self._makeOne() -- cgit v1.2.3 From 312aa1b996f786dc18d8189a7a6d9813ad609645 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Mon, 14 Dec 2015 02:26:32 -0800 Subject: - Remove broken integration test example from testing and source file, per #2172 - Update functional test with explicit instructions and to sync with actual starter scaffold --- docs/narr/MyProject/myproject/tests.py | 26 ------------ docs/narr/testing.rst | 73 +++++++++++++++++++--------------- 2 files changed, 42 insertions(+), 57 deletions(-) diff --git a/docs/narr/MyProject/myproject/tests.py b/docs/narr/MyProject/myproject/tests.py index 63d00910c..37df08a2a 100644 --- a/docs/narr/MyProject/myproject/tests.py +++ b/docs/narr/MyProject/myproject/tests.py @@ -17,32 +17,6 @@ class ViewTests(unittest.TestCase): self.assertEqual(info['project'], 'MyProject') -class ViewIntegrationTests(unittest.TestCase): - def setUp(self): - """ This sets up the application registry with the - registrations your application declares in its ``includeme`` - function. - """ - self.config = testing.setUp() - self.config.include('myproject') - - def tearDown(self): - """ Clear out the application registry """ - testing.tearDown() - - def test_my_view(self): - from myproject.views import my_view - request = testing.DummyRequest() - result = my_view(request) - self.assertEqual(result.status, '200 OK') - body = result.app_iter[0] - self.assertTrue('Welcome to' in body) - self.assertEqual(len(result.headerlist), 2) - self.assertEqual(result.headerlist[0], - ('Content-Type', 'text/html; charset=UTF-8')) - self.assertEqual(result.headerlist[1], ('Content-Length', - str(len(body)))) - class FunctionalTests(unittest.TestCase): def setUp(self): from myproject import main diff --git a/docs/narr/testing.rst b/docs/narr/testing.rst index c05ee41ad..a3f62058b 100644 --- a/docs/narr/testing.rst +++ b/docs/narr/testing.rst @@ -348,26 +348,6 @@ code's integration with the rest of :app:`Pyramid`. See also :ref:`including_configuration` -Let's demonstrate this by showing an integration test for a view. - -Given the following view definition, which assumes that your application's -:term:`package` name is ``myproject``, and within that :term:`package` there -exists a module ``views``, which in turn contains a :term:`view` function named -``my_view``: - - .. literalinclude:: MyProject/myproject/views.py - :linenos: - :lines: 1-6 - :language: python - -You'd then create a ``tests`` module within your ``myproject`` package, -containing the following test code: - - .. literalinclude:: MyProject/myproject/tests.py - :linenos: - :pyobject: ViewIntegrationTests - :language: python - Writing unit tests that use the :class:`~pyramid.config.Configurator` API to set up the right "mock" registrations is often preferred to creating integration tests. Unit tests will run faster (because they do less for each @@ -388,22 +368,53 @@ package, which provides APIs for invoking HTTP(S) requests to your application. Regardless of which testing :term:`package` you use, ensure to add a ``tests_require`` dependency on that package to your application's -``setup.py`` file: +``setup.py`` file. Using the project ``MyProject`` generated by the starter +scaffold as described in :doc:`project`, we would insert the following code immediately following the +``requires`` block in the file ``MyProject/setup.py``. - .. literalinclude:: MyProject/setup.py - :linenos: - :emphasize-lines: 26-28,48 - :language: python +.. code-block:: ini + :linenos: + :lineno-start: 11 + :emphasize-lines: 8- + + requires = [ + 'pyramid', + 'pyramid_chameleon', + 'pyramid_debugtoolbar', + 'waitress', + ] + + test_requires = [ + 'webtest', + ] + +Remember to change the dependency. + +.. code-block:: ini + :linenos: + :lineno-start: 39 + :emphasize-lines: 2 + + install_requires=requires, + tests_require=test_requires, + test_suite="myproject", + +As always, whenever you change your dependencies, make sure to run the +following command. + +.. code-block:: bash + + $VENV/bin/python setup.py develop -Let us assume your :term:`package` is named ``myproject`` which contains a -``views`` module, which in turn contains a :term:`view` function ``my_view`` -that returns a HTML body when the root URL is invoked: +In your ``MyPackage`` project, your :term:`package` is named ``myproject`` +which contains a ``views`` module, which in turn contains a :term:`view` +function ``my_view`` that returns an HTML body when the root URL is invoked: .. literalinclude:: MyProject/myproject/views.py :linenos: :language: python -Then the following example functional test demonstrates invoking the above +The following example functional test demonstrates invoking the above :term:`view`: .. literalinclude:: MyProject/myproject/tests.py @@ -414,9 +425,9 @@ Then the following example functional test demonstrates invoking the above When this test is run, each test method creates a "real" :term:`WSGI` application using the ``main`` function in your ``myproject.__init__`` module, using :term:`WebTest` to wrap that WSGI application. It assigns the result to -``self.testapp``. In the test named ``test_root``. The ``TestApp``'s ``GET`` +``self.testapp``. In the test named ``test_root``, the ``TestApp``'s ``GET`` method is used to invoke the root URL. Finally, an assertion is made that the -returned HTML contains the text ``MyProject``. +returned HTML contains the text ``Pyramid``. See the :term:`WebTest` documentation for further information about the methods available to a :class:`webtest.app.TestApp` instance. -- cgit v1.2.3 From 93d2aa696288098b046915bb718604ee9a4d7de7 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 16 Dec 2015 16:26:20 -0600 Subject: test travis irc notifications --- .travis.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.travis.yml b/.travis.yml index 2163eb8fd..6cc2e9ad4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -36,3 +36,8 @@ script: notifications: email: - pyramid-checkins@lists.repoze.org + irc: + channels: + - "chat.freenode.net#pyramid" + template: + - "%{repository}/%{branch} (%{commit} - %{author}): %{message}" -- cgit v1.2.3 From e40ccc3b4cc54fdef4fa49a131a95cd9f5bed80e Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 16 Dec 2015 16:30:39 -0600 Subject: use default irc template --- .travis.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 6cc2e9ad4..5c53b43f6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -39,5 +39,3 @@ notifications: irc: channels: - "chat.freenode.net#pyramid" - template: - - "%{repository}/%{branch} (%{commit} - %{author}): %{message}" -- cgit v1.2.3 From ca573ea73db05d7ea9bdbb51eb7db26ab602ccf2 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 16 Dec 2015 19:50:13 -0600 Subject: defer prevent check until register --- pyramid/config/views.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pyramid/config/views.py b/pyramid/config/views.py index 759276351..acaf9462b 100644 --- a/pyramid/config/views.py +++ b/pyramid/config/views.py @@ -2102,9 +2102,6 @@ class StaticURLInfo(object): config.action(None, callable=register, introspectables=(intr,)) def add_cache_buster(self, config, spec, cachebust, explicit=False): - if config.registry.settings.get('pyramid.prevent_cachebust'): - return - # ensure the spec always has a trailing slash as we only support # adding cache busters to folders, not files if os.path.isabs(spec): # FBO windows @@ -2115,6 +2112,9 @@ class StaticURLInfo(object): spec = spec + sep def register(): + if config.registry.settings.get('pyramid.prevent_cachebust'): + return + cache_busters = self.cache_busters # find duplicate cache buster (old_idx) @@ -2138,6 +2138,7 @@ class StaticURLInfo(object): if old_idx is not None: cache_busters.pop(old_idx) + cache_busters.insert(new_idx, (spec, cachebust, explicit)) intr = config.introspectable('cache busters', -- cgit v1.2.3 From 313ff3c28fbd3b784e4c87daf8ae8a4cf713262b Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Wed, 16 Dec 2015 21:30:56 -0600 Subject: update docs to support explicit asset overrides --- docs/narr/assets.rst | 100 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 35 deletions(-) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 0e3f6af11..b28e6b5b3 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -380,11 +380,6 @@ assets using :meth:`~pyramid.config.Configurator.add_cache_buster`: 'mypackage:folder/static/', QueryStringConstantCacheBuster(str(int(time.time())))) -.. note:: - The trailing slash on the ``add_cache_buster`` call is important to - indicate that it is overriding every asset in the folder and not just a - file named ``static``. - Adding the cachebuster instructs :app:`Pyramid` to add the current time for a static asset to the query string in the asset's URL: @@ -451,12 +446,13 @@ the hash of the current commit: from an egg repository like PYPI, you can use this cachebuster to get the current commit's SHA1 to use as the cache bust token. """ - def __init__(self, param='x'): + def __init__(self, param='x', repo_path=None): super(GitCacheBuster, self).__init__(param=param) - here = os.path.dirname(os.path.abspath(__file__)) + if repo_path is None: + repo_path = os.path.dirname(os.path.abspath(__file__)) self.sha1 = subprocess.check_output( ['git', 'rev-parse', 'HEAD'], - cwd=here).strip() + cwd=repo_path).strip() def tokenize(self, pathspec): return self.sha1 @@ -469,10 +465,14 @@ well: import posixpath - def cache_buster(spec, subpath, kw): - base_subpath, ext = posixpath.splitext(subpath) - new_subpath = base_subpath + '-asdf' + ext - return new_subpath, kw + class PathConstantCacheBuster(object): + def __init__(self, token): + self.token = token + + def __call__(self, request, subpath, kw): + base_subpath, ext = posixpath.splitext(subpath) + new_subpath = base_subpath + self.token + ext + return new_subpath, kw The caveat with this approach is that modifying the path segment changes the file name, and thus must match what is actually on the @@ -532,29 +532,6 @@ The following code would set up a cachebuster: 'mypackage:static/', ManifestCacheBuster('myapp:static/manifest.json')) -A simpler approach is to use the -:class:`~pyramid.static.QueryStringConstantCacheBuster` to generate a global -token that will bust all of the assets at once. The advantage of this strategy -is that it is simple and by using the query string there doesn't need to be -any shared information between your application and the static assets. - -The following code would set up a cachebuster that just uses the time at -start up as a cachebust token: - -.. code-block:: python - :linenos: - - import time - from pyramid.static import QueryStringConstantCacheBuster - - config.add_static_view( - name='http://mycdn.example.com/', - path='mypackage:static') - - config.add_cache_buster( - 'mypackage:static/', - QueryStringConstantCacheBuster(str(int(time.time())))) - .. index:: single: static assets view @@ -834,3 +811,56 @@ when an override is used. As of Pyramid 1.6, it is also possible to override an asset by supplying an absolute path to a file or directory. This may be useful if the assets are not distributed as part of a Python package. + +Cache Busting and Asset Overrides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Overriding static assets that are being hosted using +:meth:`pyramid.config.Configurator.add_static_view` can affect your cache +busting strategy when using any cache busters that are asset-aware such as +:class:`pyramid.static.ManifestCacheBuster`. What sets asset-aware cache +busters apart is that they have logic tied to specific assets. For example, +a manifest is only generated for a specific set of pre-defined assets. Now, +imagine you have overridden an asset defined in this manifest with a new, +unknown version. By default, the cache buster will be invoked for an asset +it has never seen before and will likely end up returning a cache busting +token for the original asset rather than the asset that will actually end up +being served! In order to get around this issue it's possible to attach a +different :class:`pyramid.interfaces.ICacheBuster` implementation to the +new assets. This would cause the original assets to be served by their +manifest, and the new assets served by their own cache buster. To do this, +:meth:`pyramid.config.Configurator.add_cache_buster` supports an ``explicit`` +option. For example: + +.. code-block:: python + :linenos: + + from pyramid.static import ManifestCacheBuster + + # define a static view for myapp:static assets + config.add_static_view('static', 'myapp:static') + + # setup a cache buster for your app based on the myapp:static assets + my_cb = ManifestCacheBuster('myapp:static/manifest.json') + config.add_cache_buster('myapp:static', my_cb) + + # override an asset + config.override_asset( + to_override='myapp:static/background.png', + override_with='theme:static/background.png') + + # override the cache buster for theme:static assets + theme_cb = ManifestCacheBuster('theme:static/manifest.json') + config.add_cache_buster('theme:static', theme_cb, explicit=True) + +In the above example there is a default cache buster, ``my_cb`` for all assets +served from the ``myapp:static`` folder. This would also affect +``theme:static/background.png`` when generating URLs via +``request.static_url('myapp:static/background.png')``. + +The ``theme_cb`` is defined explicitly for any assets loaded from the +``theme:static`` folder. Explicit cache busters have priority and thus +``theme_cb`` would be invoked for +``request.static_url('myapp:static/background.png')`` but ``my_cb`` would be +used for any other assets like +``request.static_url('myapp:static/favicon.ico')``. -- cgit v1.2.3 From 1dea1477ef7b06fbc1a5de4b434f6b8e6a9d9905 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Thu, 17 Dec 2015 00:11:35 -0800 Subject: progress through "Pyramid Uses a ZCA Registry" - minor grammar, wrap 79 columns --- docs/designdefense.rst | 142 ++++++++++++++++++++++++------------------------- 1 file changed, 69 insertions(+), 73 deletions(-) diff --git a/docs/designdefense.rst b/docs/designdefense.rst index ee6d5a317..478289c2b 100644 --- a/docs/designdefense.rst +++ b/docs/designdefense.rst @@ -7,98 +7,94 @@ From time to time, challenges to various aspects of :app:`Pyramid` design are lodged. To give context to discussions that follow, we detail some of the design decisions and trade-offs here. In some cases, we acknowledge that the framework can be made better and we describe future steps which will be taken -to improve it; in some cases we just file the challenge as noted, as -obviously you can't please everyone all of the time. +to improve it. In others we just file the challenge as noted, as obviously you +can't please everyone all of the time. Pyramid Provides More Than One Way to Do It ------------------------------------------- A canon of Python popular culture is "TIOOWTDI" ("there is only one way to do -it", a slighting, tongue-in-cheek reference to Perl's "TIMTOWTDI", which is -an acronym for "there is more than one way to do it"). - -:app:`Pyramid` is, for better or worse, a "TIMTOWTDI" system. For example, -it includes more than one way to resolve a URL to a :term:`view callable`: -via :term:`url dispatch` or :term:`traversal`. Multiple methods of -configuration exist: :term:`imperative configuration`, :term:`configuration -decoration`, and :term:`ZCML` (optionally via :term:`pyramid_zcml`). It works -with multiple different kinds of persistence and templating systems. And so -on. However, the existence of most of these overlapping ways to do things -are not without reason and purpose: we have a number of audiences to serve, -and we believe that TIMTOWTI at the web framework level actually *prevents* a -much more insidious and harmful set of duplication at higher levels in the -Python web community. - -:app:`Pyramid` began its life as :mod:`repoze.bfg`, written by a team of -people with many years of prior :term:`Zope` experience. The idea of +it", a slighting, tongue-in-cheek reference to Perl's "TIMTOWTDI", which is an +acronym for "there is more than one way to do it"). + +:app:`Pyramid` is, for better or worse, a "TIMTOWTDI" system. For example, it +includes more than one way to resolve a URL to a :term:`view callable`: via +:term:`url dispatch` or :term:`traversal`. Multiple methods of configuration +exist: :term:`imperative configuration`, :term:`configuration decoration`, and +:term:`ZCML` (optionally via :term:`pyramid_zcml`). It works with multiple +different kinds of persistence and templating systems. And so on. However, the +existence of most of these overlapping ways to do things are not without reason +and purpose: we have a number of audiences to serve, and we believe that +TIMTOWTDI at the web framework level actually *prevents* a much more insidious +and harmful set of duplication at higher levels in the Python web community. + +:app:`Pyramid` began its life as :mod:`repoze.bfg`, written by a team of people +with many years of prior :term:`Zope` experience. The idea of :term:`traversal` and the way :term:`view lookup` works was stolen entirely from Zope. The authorization subsystem provided by :app:`Pyramid` is a derivative of Zope's. The idea that an application can be *extended* without forking is also a Zope derivative. Implementations of these features were *required* to allow the :app:`Pyramid` -authors to build the bread-and-butter CMS-type systems for customers in the -way in which they were accustomed. No other system, save for Zope itself, -had such features, and Zope itself was beginning to show signs of its age. -We were becoming hampered by consequences of its early design mistakes. -Zope's lack of documentation was also difficult to work around: it was hard -to hire smart people to work on Zope applications, because there was no -comprehensive documentation set to point them at which explained "it all" in -one consumable place, and it was too large and self-inconsistent to document -properly. Before :mod:`repoze.bfg` went under development, its authors -obviously looked around for other frameworks that fit the bill. But no -non-Zope framework did. So we embarked on building :mod:`repoze.bfg`. +authors to build the bread-and-butter CMS-type systems for customers in the way +in which they were accustomed. No other system, save for Zope itself, had such +features, and Zope itself was beginning to show signs of its age. We were +becoming hampered by consequences of its early design mistakes. Zope's lack of +documentation was also difficult to work around. It was hard to hire smart +people to work on Zope applications because there was no comprehensive +documentation set which explained "it all" in one consumable place, and it was +too large and self-inconsistent to document properly. Before :mod:`repoze.bfg` +went under development, its authors obviously looked around for other +frameworks that fit the bill. But no non-Zope framework did. So we embarked on +building :mod:`repoze.bfg`. As the result of our research, however, it became apparent that, despite the -fact that no *one* framework had all the features we required, lots of -existing frameworks had good, and sometimes very compelling ideas. In -particular, :term:`URL dispatch` is a more direct mechanism to map URLs to -code. +fact that no *one* framework had all the features we required, lots of existing +frameworks had good, and sometimes very compelling ideas. In particular, +:term:`URL dispatch` is a more direct mechanism to map URLs to code. So, although we couldn't find a framework, save for Zope, that fit our needs, and while we incorporated a lot of Zope ideas into BFG, we also emulated the features we found compelling in other frameworks (such as :term:`url -dispatch`). After the initial public release of BFG, as time went on, -features were added to support people allergic to various Zope-isms in the -system, such as the ability to configure the application using -:term:`imperative configuration` and :term:`configuration decoration` rather -than solely using :term:`ZCML`, and the elimination of the required use of -:term:`interface` objects. It soon became clear that we had a system that -was very generic, and was beginning to appeal to non-Zope users as well as -ex-Zope users. +dispatch`). After the initial public release of BFG, as time went on, features +were added to support people allergic to various Zope-isms in the system, such +as the ability to configure the application using :term:`imperative +configuration` and :term:`configuration decoration`, rather than solely using +:term:`ZCML`, and the elimination of the required use of :term:`interface` +objects. It soon became clear that we had a system that was very generic, and +was beginning to appeal to non-Zope users as well as ex-Zope users. As the result of this generalization, it became obvious BFG shared 90% of its -featureset with the featureset of Pylons 1, and thus had a very similar -target market. Because they were so similar, choosing between the two -systems was an exercise in frustration for an otherwise non-partisan -developer. It was also strange for the Pylons and BFG development -communities to be in competition for the same set of users, given how similar -the two frameworks were. So the Pylons and BFG teams began to work together -to form a plan to merge. The features missing from BFG (notably :term:`view -handler` classes, flash messaging, and other minor missing bits), were added, -to provide familiarity to ex-Pylons users. The result is :app:`Pyramid`. - -The Python web framework space is currently notoriously balkanized. We're -truly hoping that the amalgamation of components in :app:`Pyramid` will -appeal to at least two currently very distinct sets of users: Pylons and BFG -users. By unifying the best concepts from Pylons and BFG into a single -codebase and leaving the bad concepts from their ancestors behind, we'll be -able to consolidate our efforts better, share more code, and promote our -efforts as a unit rather than competing pointlessly. We hope to be able to -shortcut the pack mentality which results in a *much larger* duplication of -effort, represented by competing but incredibly similar applications and -libraries, each built upon a specific low level stack that is incompatible -with the other. We'll also shrink the choice of credible Python web -frameworks down by at least one. We're also hoping to attract users from -other communities (such as Zope's and TurboGears') by providing the features -they require, while allowing enough flexibility to do things in a familiar -fashion. Some overlap of functionality to achieve these goals is expected -and unavoidable, at least if we aim to prevent pointless duplication at -higher levels. If we've done our job well enough, the various audiences will -be able to coexist and cooperate rather than firing at each other across some -imaginary web framework DMZ. - -Pyramid Uses A Zope Component Architecture ("ZCA") Registry +feature set with the feature set of Pylons 1, and thus had a very similar +target market. Because they were so similar, choosing between the two systems +was an exercise in frustration for an otherwise non-partisan developer. It was +also strange for the Pylons and BFG development communities to be in +competition for the same set of users, given how similar the two frameworks +were. So the Pylons and BFG teams began to work together to form a plan to +merge. The features missing from BFG (notably :term:`view handler` classes, +flash messaging, and other minor missing bits), were added to provide +familiarity to ex-Pylons users. The result is :app:`Pyramid`. + +The Python web framework space is currently notoriously balkanized. We're truly +hoping that the amalgamation of components in :app:`Pyramid` will appeal to at +least two currently very distinct sets of users: Pylons and BFG users. By +unifying the best concepts from Pylons and BFG into a single codebase, and +leaving the bad concepts from their ancestors behind, we'll be able to +consolidate our efforts better, share more code, and promote our efforts as a +unit rather than competing pointlessly. We hope to be able to shortcut the pack +mentality which results in a *much larger* duplication of effort, represented +by competing but incredibly similar applications and libraries, each built upon +a specific low level stack that is incompatible with the other. We'll also +shrink the choice of credible Python web frameworks down by at least one. We're +also hoping to attract users from other communities (such as Zope's and +TurboGears') by providing the features they require, while allowing enough +flexibility to do things in a familiar fashion. Some overlap of functionality +to achieve these goals is expected and unavoidable, at least if we aim to +prevent pointless duplication at higher levels. If we've done our job well +enough, the various audiences will be able to coexist and cooperate rather than +firing at each other across some imaginary web framework DMZ. + +Pyramid Uses a Zope Component Architecture ("ZCA") Registry ----------------------------------------------------------- :app:`Pyramid` uses a :term:`Zope Component Architecture` (ZCA) "component -- cgit v1.2.3 From 897d1deeab710233565f97d216e4d112b2a466e3 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 17 Dec 2015 20:52:19 -0600 Subject: grammar updates from stevepiercy --- docs/narr/assets.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index b28e6b5b3..054c58247 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -557,7 +557,7 @@ use some of the following options to get started: * Configure your asset pipeline to rewrite URL references inline in CSS and JavaScript. This is the best approach because then the files - may be hosted by :app:`Pyramid` or an external CDN without haven't to + may be hosted by :app:`Pyramid` or an external CDN without having to change anything. They really are static. * Templatize JS and CSS, and call ``request.static_url()`` inside their @@ -825,7 +825,7 @@ imagine you have overridden an asset defined in this manifest with a new, unknown version. By default, the cache buster will be invoked for an asset it has never seen before and will likely end up returning a cache busting token for the original asset rather than the asset that will actually end up -being served! In order to get around this issue it's possible to attach a +being served! In order to get around this issue, it's possible to attach a different :class:`pyramid.interfaces.ICacheBuster` implementation to the new assets. This would cause the original assets to be served by their manifest, and the new assets served by their own cache buster. To do this, @@ -853,14 +853,14 @@ option. For example: theme_cb = ManifestCacheBuster('theme:static/manifest.json') config.add_cache_buster('theme:static', theme_cb, explicit=True) -In the above example there is a default cache buster, ``my_cb`` for all assets -served from the ``myapp:static`` folder. This would also affect +In the above example there is a default cache buster, ``my_cb``, for all +assets served from the ``myapp:static`` folder. This would also affect ``theme:static/background.png`` when generating URLs via ``request.static_url('myapp:static/background.png')``. The ``theme_cb`` is defined explicitly for any assets loaded from the ``theme:static`` folder. Explicit cache busters have priority and thus ``theme_cb`` would be invoked for -``request.static_url('myapp:static/background.png')`` but ``my_cb`` would be -used for any other assets like +``request.static_url('myapp:static/background.png')``, but ``my_cb`` would +be used for any other assets like ``request.static_url('myapp:static/favicon.ico')``. -- cgit v1.2.3 From bc8fa64a416ce52ec5cc9fd819ce1a3caa427a19 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 17 Dec 2015 21:18:34 -0600 Subject: update changelog for #2171 --- CHANGES.txt | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index e9dc975a7..3c3dd6e79 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -53,11 +53,12 @@ Features See https://github.com/Pylons/pyramid/pull/1471 - Cache busting for static resources has been added and is available via a new - argument to ``pyramid.config.Configurator.add_static_view``: ``cachebust``. - Core APIs are shipped for both cache busting via query strings and - path segments and may be extended to fit into custom asset pipelines. + ``pyramid.config.Configurator.add_cache_buster`` API. Core APIs are shipped + for both cache busting via query strings and via asset manifests for + integrating into custom asset pipelines. See https://github.com/Pylons/pyramid/pull/1380 and - https://github.com/Pylons/pyramid/pull/1583 + https://github.com/Pylons/pyramid/pull/1583 and + https://github.com/Pylons/pyramid/pull/2171 - Add ``pyramid.config.Configurator.root_package`` attribute and init parameter to assist with includeable packages that wish to resolve -- cgit v1.2.3 From f7171cc18d2452797e1497158bed21d779c54355 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 17 Dec 2015 21:53:34 -0600 Subject: ensure IAssetDescriptor.abspath always returns an abspath fixes #2184 --- pyramid/path.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pyramid/path.py b/pyramid/path.py index b79c5a6ac..ceb0a0cf3 100644 --- a/pyramid/path.py +++ b/pyramid/path.py @@ -396,7 +396,8 @@ class PkgResourcesAssetDescriptor(object): return '%s:%s' % (self.pkg_name, self.path) def abspath(self): - return self.pkg_resources.resource_filename(self.pkg_name, self.path) + return os.path.abspath( + self.pkg_resources.resource_filename(self.pkg_name, self.path)) def stream(self): return self.pkg_resources.resource_stream(self.pkg_name, self.path) -- cgit v1.2.3 From 010218070ef62a31e3880acf2994ea217797332a Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Thu, 17 Dec 2015 22:19:46 -0600 Subject: add changelog for #2187 --- CHANGES.txt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 3c3dd6e79..0f88a95da 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -252,6 +252,11 @@ Bug Fixes process failed to fork because it could not find the pserve script to run. See https://github.com/Pylons/pyramid/pull/2137 +- Ensure that ``IAssetDescriptor.abspath`` always returns an absolute path. + There were cases depending on the process CWD that a relative path would + be returned. See https://github.com/Pylons/pyramid/issues/2187 + + Deprecations ------------ -- cgit v1.2.3 From 43fb2c233a6ebbf5c26cf66a0b1ddb16d89a1026 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Fri, 18 Dec 2015 09:50:52 -0600 Subject: deprecate pserve --user and --group options This completes the deprecation of all process management options from pserve. --- pyramid/scripts/pserve.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/pyramid/scripts/pserve.py b/pyramid/scripts/pserve.py index 95752a3be..5aaaffec9 100644 --- a/pyramid/scripts/pserve.py +++ b/pyramid/scripts/pserve.py @@ -212,8 +212,9 @@ class PServeCommand(object): self.options.set_user = self.options.set_group = None # @@: Is this the right stage to set the user at? - self.change_user_group( - self.options.set_user, self.options.set_group) + if self.options.set_user or self.options.set_group: + self.change_user_group( + self.options.set_user, self.options.set_group) if not self.args: self.out('You must give a config file') @@ -624,11 +625,16 @@ a real process manager for your processes like Systemd, Circus, or Supervisor. self.out('%s %s %s' % ('-' * 20, 'Restarting', '-' * 20)) def change_user_group(self, user, group): # pragma: no cover - if not user and not group: - return import pwd import grp + self.out('''\ +The --user and --group options have been deprecated in Pyramid 1.6. They will +be removed in a future release per Pyramid's deprecation policy. Please +consider using a real process manager for your processes like Systemd, Circus, +or Supervisor, all of which support process security. +''') + uid = gid = None if group: try: -- cgit v1.2.3 From 7593b05e4b4bd0bc85c5f46964b4e4a55286ad49 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sat, 19 Dec 2015 00:15:28 -0600 Subject: update changelog for #2189 --- CHANGES.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGES.txt b/CHANGES.txt index 0f88a95da..32c4995b8 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -263,7 +263,10 @@ Deprecations - The ``pserve`` command's daemonization features have been deprecated as well as ``--monitor-restart``. This includes the ``[start,stop,restart,status]`` subcommands as well as the ``--daemon``, ``--stop-daemon``, ``--pid-file``, - and ``--status`` flags. + ``--status``, ``--user`` and ``--group`` flags. + See https://github.com/Pylons/pyramid/pull/2120 + and https://github.com/Pylons/pyramid/pull/2189 + and https://github.com/Pylons/pyramid/pull/1641 Please use a real process manager in the future instead of relying on the ``pserve`` to daemonize itself. Many options exist including your Operating -- cgit v1.2.3 From 50dd2e4c7d8f5ab8de79c490e304b44916183f77 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Sun, 20 Dec 2015 14:00:12 -0800 Subject: add sphinxcontrib-programoutput configuration to render command line output --- docs/conf.py | 9 +++++---- setup.py | 1 + 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 8a9bac6ed..073811eca 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -53,12 +53,13 @@ extensions = [ 'sphinx.ext.doctest', 'repoze.sphinx.autointerface', 'sphinx.ext.viewcode', - 'sphinx.ext.intersphinx' + 'sphinx.ext.intersphinx', + 'sphinxcontrib.programoutput', ] # Looks for objects in external projects intersphinx_mapping = { - 'colander': ( 'http://docs.pylonsproject.org/projects/colander/en/latest', None), + 'colander': ('http://docs.pylonsproject.org/projects/colander/en/latest', None), 'cookbook': ('http://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/', None), 'deform': ('http://docs.pylonsproject.org/projects/deform/en/latest', None), 'jinja2': ('http://docs.pylonsproject.org/projects/pyramid-jinja2/en/latest/', None), @@ -66,8 +67,8 @@ intersphinx_mapping = { 'python': ('http://docs.python.org', None), 'python3': ('http://docs.python.org/3', None), 'sqla': ('http://docs.sqlalchemy.org/en/latest', None), - 'tm': ('http://docs.pylonsproject.org/projects/pyramid_tm/en/latest/', None), - 'toolbar': ('http://docs.pylonsproject.org/projects/pyramid-debugtoolbar/en/latest', None), + 'tm': ('http://docs.pylonsproject.org/projects/pyramid_tm/en/latest/', None), + 'toolbar': ('http://docs.pylonsproject.org/projects/pyramid-debugtoolbar/en/latest', None), 'tstring': ('http://docs.pylonsproject.org/projects/translationstring/en/latest', None), 'tutorials': ('http://docs.pylonsproject.org/projects/pyramid-tutorials/en/latest/', None), 'venusian': ('http://docs.pylonsproject.org/projects/venusian/en/latest', None), diff --git a/setup.py b/setup.py index c81956e7f..9bdfcd90e 100644 --- a/setup.py +++ b/setup.py @@ -61,6 +61,7 @@ docs_extras = [ 'repoze.sphinx.autointerface', 'pylons_sphinx_latesturl', 'pylons-sphinx-themes', + 'sphinxcontrib-programoutput', ] testing_extras = tests_require + [ -- cgit v1.2.3 From 5ff3d2dfdbf936d115e3486696401ad7dbffedc3 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Mon, 21 Dec 2015 01:24:34 -0800 Subject: - add p* scripts - add links to p* scripts - add a blank line to keep indices and labels better visually related to the subsequent heading --- docs/index.rst | 13 +++++++++++++ docs/narr/commandline.rst | 33 +++++++++++++++++++++++++++++++++ docs/narr/project.rst | 6 ++++++ docs/pscripts/index.rst | 12 ++++++++++++ docs/pscripts/pcreate.rst | 14 ++++++++++++++ docs/pscripts/pdistreport.rst | 14 ++++++++++++++ docs/pscripts/prequest.rst | 14 ++++++++++++++ docs/pscripts/proutes.rst | 14 ++++++++++++++ docs/pscripts/pserve.rst | 14 ++++++++++++++ docs/pscripts/pshell.rst | 14 ++++++++++++++ docs/pscripts/ptweens.rst | 14 ++++++++++++++ docs/pscripts/pviews.rst | 14 ++++++++++++++ 12 files changed, 176 insertions(+) create mode 100644 docs/pscripts/index.rst create mode 100644 docs/pscripts/pcreate.rst create mode 100644 docs/pscripts/pdistreport.rst create mode 100644 docs/pscripts/prequest.rst create mode 100644 docs/pscripts/proutes.rst create mode 100644 docs/pscripts/pserve.rst create mode 100644 docs/pscripts/pshell.rst create mode 100644 docs/pscripts/ptweens.rst create mode 100644 docs/pscripts/pviews.rst diff --git a/docs/index.rst b/docs/index.rst index e792b9905..8c8a0a18d 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -162,6 +162,19 @@ Comprehensive reference material for every public API exposed by api/* +``p*`` Scripts Documentation +============================ + +``p*`` scripts included with :app:`Pyramid`:. + +.. toctree:: + :maxdepth: 1 + :glob: + + pscripts/index + pscripts/* + + Change History ============== diff --git a/docs/narr/commandline.rst b/docs/narr/commandline.rst index eb79dffb6..34b12e1e9 100644 --- a/docs/narr/commandline.rst +++ b/docs/narr/commandline.rst @@ -6,6 +6,7 @@ Command-Line Pyramid Your :app:`Pyramid` application can be controlled and inspected using a variety of command-line utilities. These utilities are documented in this chapter. + .. index:: pair: matching views; printing single: pviews @@ -15,6 +16,8 @@ of command-line utilities. These utilities are documented in this chapter. Displaying Matching Views for a Given URL ----------------------------------------- +.. seealso:: See also the output of :ref:`pviews --help `. + For a big application with several views, it can be hard to keep the view configuration details in your head, even if you defined all the views yourself. You can use the ``pviews`` command in a terminal window to print a summary of @@ -114,6 +117,8 @@ found* message. The Interactive Shell --------------------- +.. seealso:: See also the output of :ref:`pshell --help `. + Once you've installed your program for development using ``setup.py develop``, you can use an interactive Python shell to execute expressions in a Python environment exactly like the one that will be used when your application runs @@ -179,6 +184,7 @@ hash after the filename: Press ``Ctrl-D`` to exit the interactive shell (or ``Ctrl-Z`` on Windows). + .. index:: pair: pshell; extending @@ -261,6 +267,7 @@ request is configured to generate urls from the host >>> request.route_url('home') 'https://www.example.com/' + .. _ipython_or_bpython: Alternative Shells @@ -317,6 +324,7 @@ arguments, ``env`` and ``help``, which would look like this: ``ipython`` and ``bpython`` have been moved into their respective packages ``pyramid_ipython`` and ``pyramid_bpython``. + Setting a Default Shell ~~~~~~~~~~~~~~~~~~~~~~~ @@ -331,6 +339,7 @@ specify a list of preferred shells. .. versionadded:: 1.6 + .. index:: pair: routes; printing single: proutes @@ -340,6 +349,8 @@ specify a list of preferred shells. Displaying All Application Routes --------------------------------- +.. seealso:: See also the output of :ref:`proutes --help `. + You can use the ``proutes`` command in a terminal window to print a summary of routes related to your application. Much like the ``pshell`` command (see :ref:`interactive_shell`), the ``proutes`` command accepts one argument with @@ -421,6 +432,8 @@ include. The current available formats are ``name``, ``pattern``, ``view``, and Displaying "Tweens" ------------------- +.. seealso:: See also the output of :ref:`ptweens --help `. + A :term:`tween` is a bit of code that sits between the main Pyramid application request handler and the WSGI application which calls it. A user can get a representation of both the implicit tween ordering (the ordering specified by @@ -497,6 +510,7 @@ used: See :ref:`registering_tweens` for more information about tweens. + .. index:: single: invoking a request single: prequest @@ -506,6 +520,8 @@ See :ref:`registering_tweens` for more information about tweens. Invoking a Request ------------------ +.. seealso:: See also the output of :ref:`prequest --help `. + You can use the ``prequest`` command-line utility to send a request to your application and see the response body without starting a server. @@ -555,6 +571,7 @@ of the ``prequest`` process is used as the ``POST`` body:: $ $VENV/bin/prequest -mPOST development.ini / < somefile + Using Custom Arguments to Python when Running ``p*`` Scripts ------------------------------------------------------------ @@ -566,11 +583,22 @@ Python interpreter at runtime. For example:: python -3 -m pyramid.scripts.pserve development.ini + +.. index:: + single: pdistreport + single: distributions, showing installed + single: showing installed distributions + +.. _showing_distributions: + Showing All Installed Distributions and Their Versions ------------------------------------------------------ .. versionadded:: 1.5 +.. seealso:: See also the output of :ref:`pdistreport --help + `. + You can use the ``pdistreport`` command to show the :app:`Pyramid` version in use, the Python version in use, and all installed versions of Python distributions in your Python environment:: @@ -590,6 +618,7 @@ pastebin when you are having problems and need someone with more familiarity with Python packaging and distribution than you have to look at your environment. + .. _writing_a_script: Writing a Script @@ -702,6 +731,7 @@ The above example specifies the ``another`` ``app``, ``pipeline``, or object present in the ``env`` dictionary returned by :func:`pyramid.paster.bootstrap` will be a :app:`Pyramid` :term:`router`. + Changing the Request ~~~~~~~~~~~~~~~~~~~~ @@ -742,6 +772,7 @@ Now you can readily use Pyramid's APIs for generating URLs: env['request'].route_url('verify', code='1337') # will return 'https://example.com/prefix/verify/1337' + Cleanup ~~~~~~~ @@ -757,6 +788,7 @@ callback: env['closer']() + Setting Up Logging ~~~~~~~~~~~~~~~~~~ @@ -773,6 +805,7 @@ use the following command: See :ref:`logging_chapter` for more information on logging within :app:`Pyramid`. + .. index:: single: console script diff --git a/docs/narr/project.rst b/docs/narr/project.rst index 4785b60c4..5103bb6b8 100644 --- a/docs/narr/project.rst +++ b/docs/narr/project.rst @@ -53,15 +53,19 @@ The included scaffolds are these: ``alchemy`` URL mapping via :term:`URL dispatch` and persistence via :term:`SQLAlchemy` + .. index:: single: creating a project single: project + single: pcreate .. _creating_a_project: Creating the Project -------------------- +.. seealso:: See also the output of :ref:`pcreate --help `. + In :ref:`installing_chapter`, you created a virtual Python environment via the ``virtualenv`` command. To start a :app:`Pyramid` :term:`project`, use the ``pcreate`` command installed within the virtualenv. We'll choose the @@ -262,6 +266,8 @@ single sample test exists. Running the Project Application ------------------------------- +.. seealso:: See also the output of :ref:`pserve --help `. + Once a project is installed for development, you can run the application it represents using the ``pserve`` command against the generated configuration file. In our case, this file is named ``development.ini``. diff --git a/docs/pscripts/index.rst b/docs/pscripts/index.rst new file mode 100644 index 000000000..1f54546d7 --- /dev/null +++ b/docs/pscripts/index.rst @@ -0,0 +1,12 @@ +.. _pscripts_documentation: + +``p*`` Scripts Documentation +============================ + +``p*`` scripts included with :app:`Pyramid`:. + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/pscripts/pcreate.rst b/docs/pscripts/pcreate.rst new file mode 100644 index 000000000..4e7f89572 --- /dev/null +++ b/docs/pscripts/pcreate.rst @@ -0,0 +1,14 @@ +.. index:: + single: pcreate; --help + +.. _pcreate_script: + +``pcreate`` +----------- + +.. program-output:: pcreate --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`creating_a_project` diff --git a/docs/pscripts/pdistreport.rst b/docs/pscripts/pdistreport.rst new file mode 100644 index 000000000..37d12d848 --- /dev/null +++ b/docs/pscripts/pdistreport.rst @@ -0,0 +1,14 @@ +.. index:: + single: pdistreport; --help + +.. _pdistreport_script: + +``pdistreport`` +--------------- + +.. program-output:: pdistreport --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`showing_distributions` diff --git a/docs/pscripts/prequest.rst b/docs/pscripts/prequest.rst new file mode 100644 index 000000000..a03d5b0e0 --- /dev/null +++ b/docs/pscripts/prequest.rst @@ -0,0 +1,14 @@ +.. index:: + single: prequest; --help + +.. _prequest_script: + +``prequest`` +------------ + +.. program-output:: prequest --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`invoking_a_request` diff --git a/docs/pscripts/proutes.rst b/docs/pscripts/proutes.rst new file mode 100644 index 000000000..8f3f34e16 --- /dev/null +++ b/docs/pscripts/proutes.rst @@ -0,0 +1,14 @@ +.. index:: + single: proutes; --help + +.. _proutes_script: + +``proutes`` +----------- + +.. program-output:: proutes --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`displaying_application_routes` diff --git a/docs/pscripts/pserve.rst b/docs/pscripts/pserve.rst new file mode 100644 index 000000000..4e41d6e2b --- /dev/null +++ b/docs/pscripts/pserve.rst @@ -0,0 +1,14 @@ +.. index:: + single: pserve; --help + +.. _pserve_script: + +``pserve`` +---------- + +.. program-output:: pserve --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`running_the_project_application` diff --git a/docs/pscripts/pshell.rst b/docs/pscripts/pshell.rst new file mode 100644 index 000000000..fd85a11a9 --- /dev/null +++ b/docs/pscripts/pshell.rst @@ -0,0 +1,14 @@ +.. index:: + single: pshell; --help + +.. _pshell_script: + +``pshell`` +---------- + +.. program-output:: pshell --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`interactive_shell` diff --git a/docs/pscripts/ptweens.rst b/docs/pscripts/ptweens.rst new file mode 100644 index 000000000..e8c0d1ad0 --- /dev/null +++ b/docs/pscripts/ptweens.rst @@ -0,0 +1,14 @@ +.. index:: + single: ptweens; --help + +.. _ptweens_script: + +``ptweens`` +----------- + +.. program-output:: ptweens --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`displaying_tweens` diff --git a/docs/pscripts/pviews.rst b/docs/pscripts/pviews.rst new file mode 100644 index 000000000..a1c2f5c2b --- /dev/null +++ b/docs/pscripts/pviews.rst @@ -0,0 +1,14 @@ +.. index:: + single: pviews; --help + +.. _pviews_script: + +``pviews`` +---------- + +.. program-output:: pviews --help + :cwd: ../../env/bin + :prompt: + :shell: + +.. seealso:: :ref:`displaying_matching_views` -- cgit v1.2.3 From d8101d6e99012bd3a7fcf1806e7d922c8d1371d9 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Mon, 21 Dec 2015 21:37:52 -0800 Subject: attempt to get travis to pass on programoutput rst directive --- docs/pscripts/pcreate.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/pscripts/pcreate.rst b/docs/pscripts/pcreate.rst index 4e7f89572..b5ec3f4e2 100644 --- a/docs/pscripts/pcreate.rst +++ b/docs/pscripts/pcreate.rst @@ -7,7 +7,6 @@ ----------- .. program-output:: pcreate --help - :cwd: ../../env/bin :prompt: :shell: -- cgit v1.2.3 From cdb9f7a18725a992f8ad0bfc920c028082222b47 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Mon, 21 Dec 2015 21:42:32 -0800 Subject: remove :cwd: argument from programoutput rst directive so that travis will pass --- docs/pscripts/pdistreport.rst | 1 - docs/pscripts/prequest.rst | 1 - docs/pscripts/proutes.rst | 1 - docs/pscripts/pserve.rst | 1 - docs/pscripts/pshell.rst | 1 - docs/pscripts/ptweens.rst | 1 - docs/pscripts/pviews.rst | 1 - 7 files changed, 7 deletions(-) diff --git a/docs/pscripts/pdistreport.rst b/docs/pscripts/pdistreport.rst index 37d12d848..1c53fb6e9 100644 --- a/docs/pscripts/pdistreport.rst +++ b/docs/pscripts/pdistreport.rst @@ -7,7 +7,6 @@ --------------- .. program-output:: pdistreport --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/prequest.rst b/docs/pscripts/prequest.rst index a03d5b0e0..a15827767 100644 --- a/docs/pscripts/prequest.rst +++ b/docs/pscripts/prequest.rst @@ -7,7 +7,6 @@ ------------ .. program-output:: prequest --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/proutes.rst b/docs/pscripts/proutes.rst index 8f3f34e16..09ed013e1 100644 --- a/docs/pscripts/proutes.rst +++ b/docs/pscripts/proutes.rst @@ -7,7 +7,6 @@ ----------- .. program-output:: proutes --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/pserve.rst b/docs/pscripts/pserve.rst index 4e41d6e2b..d33d4a484 100644 --- a/docs/pscripts/pserve.rst +++ b/docs/pscripts/pserve.rst @@ -7,7 +7,6 @@ ---------- .. program-output:: pserve --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/pshell.rst b/docs/pscripts/pshell.rst index fd85a11a9..cfd84d4f8 100644 --- a/docs/pscripts/pshell.rst +++ b/docs/pscripts/pshell.rst @@ -7,7 +7,6 @@ ---------- .. program-output:: pshell --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/ptweens.rst b/docs/pscripts/ptweens.rst index e8c0d1ad0..02e23e49a 100644 --- a/docs/pscripts/ptweens.rst +++ b/docs/pscripts/ptweens.rst @@ -7,7 +7,6 @@ ----------- .. program-output:: ptweens --help - :cwd: ../../env/bin :prompt: :shell: diff --git a/docs/pscripts/pviews.rst b/docs/pscripts/pviews.rst index a1c2f5c2b..b4de5c054 100644 --- a/docs/pscripts/pviews.rst +++ b/docs/pscripts/pviews.rst @@ -7,7 +7,6 @@ ---------- .. program-output:: pviews --help - :cwd: ../../env/bin :prompt: :shell: -- cgit v1.2.3 From f829b1c82595849e0f7f685f8c559d021748a0d2 Mon Sep 17 00:00:00 2001 From: kpinc Date: Tue, 22 Dec 2015 09:47:12 -0600 Subject: Expound. Punctuation. --- docs/pscripts/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pscripts/index.rst b/docs/pscripts/index.rst index 1f54546d7..857e0564f 100644 --- a/docs/pscripts/index.rst +++ b/docs/pscripts/index.rst @@ -3,7 +3,7 @@ ``p*`` Scripts Documentation ============================ -``p*`` scripts included with :app:`Pyramid`:. +Command line programs (``p*`` scripts) included with :app:`Pyramid`. .. toctree:: :maxdepth: 1 -- cgit v1.2.3 From 091941ddfaf2e2349b7884f15ce3d79339a8e835 Mon Sep 17 00:00:00 2001 From: kpinc Date: Tue, 22 Dec 2015 09:56:42 -0600 Subject: Update prequest.py --- pyramid/scripts/prequest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyramid/scripts/prequest.py b/pyramid/scripts/prequest.py index 61e422c64..e07f9d10e 100644 --- a/pyramid/scripts/prequest.py +++ b/pyramid/scripts/prequest.py @@ -14,7 +14,7 @@ def main(argv=sys.argv, quiet=False): class PRequestCommand(object): description = """\ - Run a request for the described application. + Submit a HTTP request to a web application. This command makes an artifical request to a web application that uses a PasteDeploy (.ini) configuration file for the server and application. -- cgit v1.2.3 From d9da2b29861d071b9fc044319421799a3d522bcc Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Wed, 23 Dec 2015 02:31:05 -0800 Subject: Add documentation of command line programs/p* scripts --- CHANGES.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGES.txt b/CHANGES.txt index 32c4995b8..77f5d94ac 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -301,6 +301,9 @@ Docs ``principal`` and a ``userid`` in its security APIs. See https://github.com/Pylons/pyramid/pull/1399 +- Add documentation of command line programs (``p*`` scripts). See + https://github.com/Pylons/pyramid/pull/2191 + Scaffolds --------- -- cgit v1.2.3 From 169dba5c7aa02db2e48cecff8b8126b767fdf327 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Wed, 23 Dec 2015 03:19:58 -0800 Subject: - Add Python compatibility to history, hacking, releasing - Use 1.X for version number --- HACKING.txt | 8 ++++---- HISTORY.txt | 4 ++++ RELEASING.txt | 14 +++++++------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/HACKING.txt b/HACKING.txt index c838fda22..5bbdce0c6 100644 --- a/HACKING.txt +++ b/HACKING.txt @@ -124,10 +124,10 @@ In order to add a feature to Pyramid: - The feature must be documented in both the API and narrative documentation (in ``docs/``). -- The feature must work fully on the following CPython versions: 2.6, - 2.7, 3.2, and 3.3 on both UNIX and Windows. +- The feature must work fully on the following CPython versions: 2.6, 2.7, 3.2, + 3.3, 3.4, and 3.5 on both UNIX and Windows. -- The feature must work on the latest version of PyPy. +- The feature must work on the latest version of PyPy and PyPy3. - The feature must not cause installation or runtime failure on App Engine. If it doesn't cause installation or runtime failure, but doesn't actually @@ -199,7 +199,7 @@ Running Tests Alternately:: - $ tox -e{py26,py27,py32,py33,py34,pypy,pypy3}-scaffolds, + $ tox -e{py26,py27,py32,py33,py34,py35,pypy,pypy3}-scaffolds, Test Coverage ------------- diff --git a/HISTORY.txt b/HISTORY.txt index c30bb2711..68ddb3a90 100644 --- a/HISTORY.txt +++ b/HISTORY.txt @@ -1,6 +1,8 @@ 1.5 (2014-04-08) ================ +- Python 3.4 compatibility. + - Avoid crash in ``pserve --reload`` under Py3k, when iterating over possibly mutated ``sys.modules``. @@ -1130,6 +1132,8 @@ Bug Fixes Features -------- +- Python 3.3 compatibility. + - Configurator.add_directive now accepts arbitrary callables like partials or objects implementing ``__call__`` which dont have ``__name__`` and ``__doc__`` attributes. See https://github.com/Pylons/pyramid/issues/621 diff --git a/RELEASING.txt b/RELEASING.txt index 87ff62c53..fa4ebab5b 100644 --- a/RELEASING.txt +++ b/RELEASING.txt @@ -15,7 +15,7 @@ Releasing Pyramid - Run tests on Windows if feasible. -- Make sure all scaffold tests pass (Py 2.6, 2.7, 3.2, 3.3, 3.4, pypy, and +- Make sure all scaffold tests pass (Py 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, pypy, and pypy3 on UNIX; this doesn't work on Windows): $ ./scaffoldtests.sh @@ -69,22 +69,22 @@ Releasing Pyramid Announcement template ---------------------- -Pyramid 1.1.X has been released. +Pyramid 1.X.X has been released. Here are the changes: <> -A "What's New In Pyramid 1.1" document exists at -http://docs.pylonsproject.org/projects/pyramid/1.1/whatsnew-1.1.html . +A "What's New In Pyramid 1.X" document exists at +http://docs.pylonsproject.org/projects/pyramid/1.X/whatsnew-1.X.html . -You will be able to see the 1.1 release documentation (across all +You will be able to see the 1.X release documentation (across all alphas and betas, as well as when it eventually gets to final release) -at http://docs.pylonsproject.org/projects/pyramid/1.1/ . +at http://docs.pylonsproject.org/projects/pyramid/1.X/ . You can install it via PyPI: - easy_install Pyramid==1.1a4 + easy_install Pyramid==1.X Enjoy, and please report any issues you find to the issue tracker at https://github.com/Pylons/pyramid/issues -- cgit v1.2.3 From b7057f3dcac875d71916d6807d157ff6f3ede662 Mon Sep 17 00:00:00 2001 From: Bastien Date: Sat, 26 Dec 2015 14:33:23 -0800 Subject: Update glossary.rst fixed typo (cherry picked from commit 0a5c9a2) --- docs/glossary.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/glossary.rst b/docs/glossary.rst index b4bb36421..60e861597 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -273,7 +273,7 @@ Glossary (Allow, 'bob', 'read'), (Deny, 'fred', 'write')]``. If an ACL is attached to a resource instance, and that resource is findable via the context resource, it will be consulted any active security policy to - determine wither a particular request can be fulfilled given the + determine whether a particular request can be fulfilled given the :term:`authentication` information in the request. authentication -- cgit v1.2.3 From e8609bb3437a4642ce33aa13ca65e84003b397ca Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Wed, 30 Dec 2015 02:08:11 -0800 Subject: - minor grammar in "problems", rewrap to 79 columns --- docs/designdefense.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/designdefense.rst b/docs/designdefense.rst index 478289c2b..bfde25246 100644 --- a/docs/designdefense.rst +++ b/docs/designdefense.rst @@ -142,7 +142,7 @@ dictionary API, but that's not very important in this context. That's problem number two. Third of all, what does the ``getUtility`` function do? It's performing a -lookup for the ``ISettings`` "utility" that should return.. well, a utility. +lookup for the ``ISettings`` "utility" that should return... well, a utility. Note how we've already built up a dependency on the understanding of an :term:`interface` and the concept of "utility" to answer this question: a bad sign so far. Note also that the answer is circular, a *really* bad sign. @@ -152,12 +152,12 @@ registry" of course. What's a component registry? Problem number four. Fifth, assuming you buy that there's some magical registry hanging around, where *is* this registry? *Homina homina*... "around"? That's sort of the -best answer in this context (a more specific answer would require knowledge -of internals). Can there be more than one registry? Yes. So *which* -registry does it find the registration in? Well, the "current" registry of -course. In terms of :app:`Pyramid`, the current registry is a thread local -variable. Using an API that consults a thread local makes understanding how -it works non-local. +best answer in this context (a more specific answer would require knowledge of +internals). Can there be more than one registry? Yes. So in *which* registry +does it find the registration? Well, the "current" registry of course. In +terms of :app:`Pyramid`, the current registry is a thread local variable. +Using an API that consults a thread local makes understanding how it works +non-local. You've now bought in to the fact that there's a registry that is just hanging around. But how does the registry get populated? Why, via code that calls @@ -166,10 +166,10 @@ registration of ``ISettings`` is made by the framework itself under the hood: it's not present in any user configuration. This is extremely hard to comprehend. Problem number six. -Clearly there's some amount of cognitive load here that needs to be borne by -a reader of code that extends the :app:`Pyramid` framework due to its use of -the ZCA, even if he or she is already an expert Python programmer and whom is -an expert in the domain of web applications. This is suboptimal. +Clearly there's some amount of cognitive load here that needs to be borne by a +reader of code that extends the :app:`Pyramid` framework due to its use of the +ZCA, even if they are already an expert Python programmer and an expert in the +domain of web applications. This is suboptimal. Ameliorations +++++++++++++ -- cgit v1.2.3 From 752d6575d7b86eeb1a7b16eac9476a7620897438 Mon Sep 17 00:00:00 2001 From: Steve Piercy Date: Sat, 2 Jan 2016 22:10:01 -0800 Subject: - minor grammar in "problems", rewrap to 79 columns --- docs/whatsnew-1.5.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/whatsnew-1.5.rst b/docs/whatsnew-1.5.rst index 1d863c937..8a769e88e 100644 --- a/docs/whatsnew-1.5.rst +++ b/docs/whatsnew-1.5.rst @@ -136,6 +136,8 @@ Feature Additions The feature additions in Pyramid 1.5 follow. +- Python 3.4 compatibility. + - Add ``pdistreport`` script, which prints the Python version in use, the Pyramid version in use, and the version number and location of all Python distributions currently installed. -- cgit v1.2.3 From 2901e968390f70b4bfa777dd4348dc9d3eb6210c Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 3 Jan 2016 00:22:44 -0600 Subject: add a note about serving cache busted assets --- docs/narr/assets.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/narr/assets.rst b/docs/narr/assets.rst index 054c58247..58f547fc9 100644 --- a/docs/narr/assets.rst +++ b/docs/narr/assets.rst @@ -532,6 +532,14 @@ The following code would set up a cachebuster: 'mypackage:static/', ManifestCacheBuster('myapp:static/manifest.json')) +It's important to note that the cache buster only handles generating +cache-busted URLs for static assets. It does **NOT** provide any solutions for +serving those assets. For example, if you generated a URL for +``css/main-678b7c80.css`` then that URL needs to be valid either by +configuring ``add_static_view`` properly to point to the location of the files +or some other mechanism such as the files existing on your CDN or rewriting +the incoming URL to remove the cache bust tokens. + .. index:: single: static assets view -- cgit v1.2.3 From 41aeac4b9442a03c961b85b896ae84a8da3dbc9c Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 3 Jan 2016 01:06:35 -0600 Subject: fix grammar on whatsnew document titles --- docs/whatsnew-1.0.rst | 2 +- docs/whatsnew-1.1.rst | 2 +- docs/whatsnew-1.2.rst | 2 +- docs/whatsnew-1.3.rst | 2 +- docs/whatsnew-1.4.rst | 2 +- docs/whatsnew-1.5.rst | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/whatsnew-1.0.rst b/docs/whatsnew-1.0.rst index 9541f0a28..0ed6e21fc 100644 --- a/docs/whatsnew-1.0.rst +++ b/docs/whatsnew-1.0.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.0 +What's New in Pyramid 1.0 ========================= This article explains the new features in Pyramid version 1.0 as compared to diff --git a/docs/whatsnew-1.1.rst b/docs/whatsnew-1.1.rst index 99737b6d8..a5c7f3393 100644 --- a/docs/whatsnew-1.1.rst +++ b/docs/whatsnew-1.1.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.1 +What's New in Pyramid 1.1 ========================= This article explains the new features in Pyramid version 1.1 as compared to diff --git a/docs/whatsnew-1.2.rst b/docs/whatsnew-1.2.rst index a9fc38908..9ff933ace 100644 --- a/docs/whatsnew-1.2.rst +++ b/docs/whatsnew-1.2.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.2 +What's New in Pyramid 1.2 ========================= This article explains the new features in :app:`Pyramid` version 1.2 as diff --git a/docs/whatsnew-1.3.rst b/docs/whatsnew-1.3.rst index 2606c3df3..1a299e126 100644 --- a/docs/whatsnew-1.3.rst +++ b/docs/whatsnew-1.3.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.3 +What's New in Pyramid 1.3 ========================= This article explains the new features in :app:`Pyramid` version 1.3 as diff --git a/docs/whatsnew-1.4.rst b/docs/whatsnew-1.4.rst index 505b9d798..fce889854 100644 --- a/docs/whatsnew-1.4.rst +++ b/docs/whatsnew-1.4.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.4 +What's New in Pyramid 1.4 ========================= This article explains the new features in :app:`Pyramid` version 1.4 as diff --git a/docs/whatsnew-1.5.rst b/docs/whatsnew-1.5.rst index 8a769e88e..a477ce5ec 100644 --- a/docs/whatsnew-1.5.rst +++ b/docs/whatsnew-1.5.rst @@ -1,4 +1,4 @@ -What's New In Pyramid 1.5 +What's New in Pyramid 1.5 ========================= This article explains the new features in :app:`Pyramid` version 1.5 as -- cgit v1.2.3 From 5558386fd1a6181b2e8ad06659049c055a7ed023 Mon Sep 17 00:00:00 2001 From: Michael Merickel Date: Sun, 3 Jan 2016 01:09:52 -0600 Subject: copy whatsnew-1.6 from 1.6-branch --- docs/whatsnew-1.6.rst | 204 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 135 insertions(+), 69 deletions(-) diff --git a/docs/whatsnew-1.6.rst b/docs/whatsnew-1.6.rst index b99ebeec4..bdfcf34ab 100644 --- a/docs/whatsnew-1.6.rst +++ b/docs/whatsnew-1.6.rst @@ -1,40 +1,62 @@ -What's New In Pyramid 1.6 +What's New in Pyramid 1.6 ========================= This article explains the new features in :app:`Pyramid` version 1.6 as -compared to its predecessor, :app:`Pyramid` 1.5. It also documents backwards +compared to its predecessor, :app:`Pyramid` 1.5. It also documents backwards incompatibilities between the two versions and deprecations added to :app:`Pyramid` 1.6, as well as software dependency changes and notable documentation additions. + Backwards Incompatibilities --------------------------- +- IPython and BPython support have been removed from pshell in the core. To + continue using them on Pyramid 1.6+, you must install the binding packages + explicitly. One way to do this is by adding ``pyramid_ipython`` (or + ``pyramid_bpython``) to the ``install_requires`` section of your package's + ``setup.py`` file, then re-running ``setup.py develop``:: + + setup( + #... + install_requires=[ + 'pyramid_ipython', # new dependency + 'pyramid', + #... + ], + ) + - ``request.response`` will no longer be mutated when using the - :func:`~pyramid.renderers.render_to_response` API. It is now necessary - to pass in - a ``response=`` argument to :func:`~pyramid.renderers.render_to_response` if - you wish to supply the renderer with a custom response object for it to - use. If you do not pass one then a response object will be created using the - current response factory. Almost all renderers mutate the - ``request.response`` response object (for example, the JSON renderer sets - ``request.response.content_type`` to ``application/json``). However, when - invoking ``render_to_response`` it is not expected that the response object - being returned would be the same one used later in the request. The response - object returned from ``render_to_response`` is now explicitly different from - ``request.response``. This does not change the API of a renderer. See + :func:`~pyramid.renderers.render_to_response` API. It is now necessary to + pass in a ``response=`` argument to + :func:`~pyramid.renderers.render_to_response` if you wish to supply the + renderer with a custom response object. If you do not pass one, then a + response object will be created using the current response factory. Almost + all renderers mutate the ``request.response`` response object (for example, + the JSON renderer sets ``request.response.content_type`` to + ``application/json``). However, when invoking ``render_to_response``, it is + not expected that the response object being returned would be the same one + used later in the request. The response object returned from + ``render_to_response`` is now explicitly different from ``request.response``. + This does not change the API of a renderer. See https://github.com/Pylons/pyramid/pull/1563 Feature Additions ----------------- -- Cache busting for static assets has been added and is available via a new - argument to :meth:`pyramid.config.Configurator.add_static_view`: - ``cachebust``. Core APIs are shipped for both cache busting via query - strings and path segments and may be extended to fit into custom asset - pipelines. See https://github.com/Pylons/pyramid/pull/1380 and - https://github.com/Pylons/pyramid/pull/1583 +- Python 3.5 and pypy3 compatibility. + +- ``pserve --reload`` will no longer crash on syntax errors. See + https://github.com/Pylons/pyramid/pull/2044 + +- Cache busting for static resources has been added and is available via a new + :meth:`pyramid.config.Configurator.add_cache_buster` API. Core APIs are + shipped for both cache busting via query strings and via asset manifests for + integrating into custom asset pipelines. See + https://github.com/Pylons/pyramid/pull/1380 and + https://github.com/Pylons/pyramid/pull/1583 and + https://github.com/Pylons/pyramid/pull/2171 - Assets can now be overidden by an absolute path on the filesystem when using the :meth:`~pyramid.config.Configurator.override_asset` API. This makes it @@ -47,99 +69,129 @@ Feature Additions ``config.add_static_view('myapp:static', 'static')`` and ``config.override_asset(to_override='myapp:static/', override_with='/abs/path/')``. The ``myapp:static`` asset spec is completely - made up and does not need to exist - it is used for generating urls via - ``request.static_url('myapp:static/foo.png')``. See + made up and does not need to exist—it is used for generating URLs via + ``request.static_url('myapp:static/foo.png')``. See https://github.com/Pylons/pyramid/issues/1252 - Added :meth:`~pyramid.config.Configurator.set_response_factory` and the ``response_factory`` keyword argument to the constructor of :class:`~pyramid.config.Configurator` for defining a factory that will return - a custom ``Response`` class. See https://github.com/Pylons/pyramid/pull/1499 + a custom ``Response`` class. See https://github.com/Pylons/pyramid/pull/1499 -- Add :attr:`pyramid.config.Configurator.root_package` attribute and init - parameter to assist with includeable packages that wish to resolve - resources relative to the package in which the configurator was created. - This is especially useful for addons that need to load asset specs from - settings, in which case it is may be natural for a developer to define - imports or assets relative to the top-level package. - See https://github.com/Pylons/pyramid/pull/1337 +- Added :attr:`pyramid.config.Configurator.root_package` attribute and init + parameter to assist with includible packages that wish to resolve resources + relative to the package in which the configurator was created. This is + especially useful for add-ons that need to load asset specs from settings, in + which case it may be natural for a developer to define imports or assets + relative to the top-level package. See + https://github.com/Pylons/pyramid/pull/1337 - Overall improvments for the ``proutes`` command. Added ``--format`` and ``--glob`` arguments to the command, introduced the ``method`` column for displaying available request methods, and improved the ``view`` - output by showing the module instead of just ``__repr__``. - See https://github.com/Pylons/pyramid/pull/1488 + output by showing the module instead of just ``__repr__``. See + https://github.com/Pylons/pyramid/pull/1488 - ``pserve`` can now take a ``-b`` or ``--browser`` option to open the server URL in a web browser. See https://github.com/Pylons/pyramid/pull/1533 -- Support keyword-only arguments and function annotations in views in - Python 3. See https://github.com/Pylons/pyramid/pull/1556 +- Support keyword-only arguments and function annotations in views in Python 3. + See https://github.com/Pylons/pyramid/pull/1556 - The ``append_slash`` argument of :meth:`~pyramid.config.Configurator.add_notfound_view()` will now accept anything that implements the :class:`~pyramid.interfaces.IResponse` interface and will use that as the response class instead of the default - :class:`~pyramid.httpexceptions.HTTPFound`. See + :class:`~pyramid.httpexceptions.HTTPFound`. See https://github.com/Pylons/pyramid/pull/1610 - The :class:`~pyramid.config.Configurator` has grown the ability to allow - actions to call other actions during a commit-cycle. This enables much more + actions to call other actions during a commit cycle. This enables much more logic to be placed into actions, such as the ability to invoke other actions or group them for improved conflict detection. We have also exposed and - documented the config phases that Pyramid uses in order to further assist in - building conforming addons. See https://github.com/Pylons/pyramid/pull/1513 + documented the configuration phases that Pyramid uses in order to further + assist in building conforming add-ons. See + https://github.com/Pylons/pyramid/pull/1513 - Allow an iterator to be returned from a renderer. Previously it was only - possible to return bytes or unicode. - See https://github.com/Pylons/pyramid/pull/1417 + possible to return bytes or unicode. See + https://github.com/Pylons/pyramid/pull/1417 - Improve robustness to timing attacks in the :class:`~pyramid.authentication.AuthTktCookieHelper` and the :class:`~pyramid.session.SignedCookieSessionFactory` classes by using the - stdlib's ``hmac.compare_digest`` if it is available (such as Python 2.7.7+ and - 3.3+). See https://github.com/Pylons/pyramid/pull/1457 + stdlib's ``hmac.compare_digest`` if it is available (such as Python 2.7.7+ + and 3.3+). See https://github.com/Pylons/pyramid/pull/1457 -- Improve the readability of the ``pcreate`` shell script output. - See https://github.com/Pylons/pyramid/pull/1453 +- Improve the readability of the ``pcreate`` shell script output. See + https://github.com/Pylons/pyramid/pull/1453 -- Make it simple to define notfound and forbidden views that wish to use the - default exception-response view but with altered predicates and other - configuration options. The ``view`` argument is now optional in +- Make it simple to define ``notfound`` and ``forbidden`` views that wish to + use the default exception-response view, but with altered predicates and + other configuration options. The ``view`` argument is now optional in :meth:`~pyramid.config.Configurator.add_notfound_view` and :meth:`~pyramid.config.Configurator.add_forbidden_view` See https://github.com/Pylons/pyramid/issues/494 - The ``pshell`` script will now load a ``PYTHONSTARTUP`` file if one is - defined in the environment prior to launching the interpreter. - See https://github.com/Pylons/pyramid/pull/1448 + defined in the environment prior to launching the interpreter. See + https://github.com/Pylons/pyramid/pull/1448 -- Add new HTTP exception objects for status codes - ``428 Precondition Required``, ``429 Too Many Requests`` and - ``431 Request Header Fields Too Large`` in ``pyramid.httpexceptions``. - See https://github.com/Pylons/pyramid/pull/1372/files +- Add new HTTP exception objects for status codes ``428 Precondition + Required``, ``429 Too Many Requests`` and ``431 Request Header Fields Too + Large`` in ``pyramid.httpexceptions``. See + https://github.com/Pylons/pyramid/pull/1372/files - ``pcreate`` when run without a scaffold argument will now print information - on the missing flag, as well as a list of available scaffolds. See + on the missing flag, as well as a list of available scaffolds. See https://github.com/Pylons/pyramid/pull/1566 and https://github.com/Pylons/pyramid/issues/1297 +- ``pcreate`` will now ask for confirmation if invoked with an argument for a + project name that already exists or is importable in the current environment. + See https://github.com/Pylons/pyramid/issues/1357 and + https://github.com/Pylons/pyramid/pull/1837 + - Add :func:`pyramid.request.apply_request_extensions` function which can be used in testing to apply any request extensions configured via ``config.add_request_method``. Previously it was only possible to test the - extensions by going through Pyramid's router. See + extensions by going through Pyramid's router. See https://github.com/Pylons/pyramid/pull/1581 - - Make it possible to subclass ``pyramid.request.Request`` and also use - ``pyramid.request.Request.add_request.method``. See + ``pyramid.request.Request.add_request.method``. See https://github.com/Pylons/pyramid/issues/1529 +- Additional shells for ``pshell`` can now be registered as entry points. See + https://github.com/Pylons/pyramid/pull/1891 and + https://github.com/Pylons/pyramid/pull/2012 + +- The variables injected into ``pshell`` are now displayed with their + docstrings instead of the default ``str(obj)`` when possible. See + https://github.com/Pylons/pyramid/pull/1929 + + Deprecations ------------ +- The ``pserve`` command's daemonization features, as well as + ``--monitor-restart``, have been deprecated. This includes the + ``[start,stop,restart,status]`` subcommands, as well as the ``--daemon``, + ``--stop-daemon``, ``--pid-file``, ``--status``, ``--user``, and ``--group`` + flags. See https://github.com/Pylons/pyramid/pull/2120 and + https://github.com/Pylons/pyramid/pull/2189 and + https://github.com/Pylons/pyramid/pull/1641 + + Please use a real process manager in the future instead of relying on + ``pserve`` to daemonize itself. Many options exist, including your operating + system's services, such as Systemd or Upstart, as well as Python-based + solutions like Circus and Supervisor. + + See https://github.com/Pylons/pyramid/pull/1641 and + https://github.com/Pylons/pyramid/pull/2120 + - The ``principal`` argument to :func:`pyramid.security.remember` was renamed - to ``userid``. Using ``principal`` as the argument name still works and will + to ``userid``. Using ``principal`` as the argument name still works and will continue to work for the next few releases, but a deprecation warning is printed. @@ -150,21 +202,35 @@ Scaffolding Enhancements - Added line numbers to the log formatters in the scaffolds to assist with debugging. See https://github.com/Pylons/pyramid/pull/1326 -- Update scaffold generating machinery to return the version of pyramid and - pyramid docs for use in scaffolds. Updated ``starter``, ``alchemy`` and - ``zodb`` templates to have links to correctly versioned documentation and - reflect which pyramid was used to generate the scaffold. +- Updated scaffold generating machinery to return the version of :app:`Pyramid` + and its documentation for use in scaffolds. Updated ``starter``, ``alchemy`` + and ``zodb`` templates to have links to correctly versioned documentation, + and to reflect which :app:`Pyramid` was used to generate the scaffold. + +- Removed non-ASCII copyright symbol from templates, as this was causing the + scaffolds to fail for project generation. -- Removed non-ascii copyright symbol from templates, as this was - causing the scaffolds to fail for project generation. Documentation Enhancements -------------------------- -- Removed logging configuration from Quick Tutorial ini files except for - scaffolding- and logging-related chapters to avoid needing to explain it too +- Removed logging configuration from Quick Tutorial ``ini`` files, except for + scaffolding- and logging-related chapters, to avoid needing to explain it too early. -- Improve and clarify the documentation on what Pyramid defines as a - ``principal`` and a ``userid`` in its security APIs. - See https://github.com/Pylons/pyramid/pull/1399 +- Improve and clarify the documentation on what :app:`Pyramid` defines as a + ``principal`` and a ``userid`` in its security APIs. See + https://github.com/Pylons/pyramid/pull/1399 + +- Moved the documentation for ``accept`` on + :meth:`pyramid.config.Configurator.add_view` to no longer be part of the + predicate list. See https://github.com/Pylons/pyramid/issues/1391 for a bug + report stating ``not_`` was failing on ``accept``. Discussion with @mcdonc + led to the conclusion that it should not be documented as a predicate. + See https://github.com/Pylons/pyramid/pull/1487 for this PR. + +- Clarify a previously-implied detail of the ``ISession.invalidate`` API + documentation. + +- Add documentation of command line programs (``p*`` scripts). See + https://github.com/Pylons/pyramid/pull/2191 -- cgit v1.2.3