security policy docs and legacy policy improvements

- Added `set_security_policy`` to more places in the docs.
- Ensure that the authn/authz policies are not used at all if the legacy
  policy is not in effect to avoid edge cases where the code would skip
  the security policy and use the authn/authz policy on accident.
- Change deprecation warnings in code to reference the docs by name
  instead of by URL.

16 files changed, 131 insertions, 68 deletions

diff --git a/docs/designdefense.rst b/docs/designdefense.rst
index 967a1aaed..0fa609aa1 100644
--- a/docs/designdefense.rst
+++ b/docs/designdefense.rst
@@ -199,11 +199,11 @@ Under its hood however, the implementation of ``authenticated_userid`` is this:
 
     def authenticated_userid(request):
         """ Return the userid of the currently authenticated user or
-        ``None`` if there is no authentication policy in effect or there
+        ``None`` if there is no security policy in effect or there
         is no currently authenticated user. """
 
         registry = request.registry # the ZCA component registry
-        policy = registry.queryUtility(IAuthenticationPolicy)
+        policy = registry.queryUtility(ISecurityPolicy)
         if policy is None:
             return None
         return policy.authenticated_userid(request)
@@ -264,19 +264,19 @@ instead of the rule.  So instead of:
 .. code-block:: python
     :linenos:
 
-    from pyramid.interfaces import IAuthenticationPolicy
+    from pyramid.interfaces import ISecurityPolicy
     from zope.component import getUtility
-    policy = getUtility(IAuthenticationPolicy)
+    policy = getUtility(ISecurityPolicy)
 
 :app:`Pyramid` code will usually do:
 
 .. code-block:: python
     :linenos:
 
-    from pyramid.interfaces import IAuthenticationPolicy
+    from pyramid.interfaces import ISecurityPolicy
     from pyramid.threadlocal import get_current_registry
     registry = get_current_registry()
-    policy = registry.getUtility(IAuthenticationPolicy)
+    policy = registry.getUtility(ISecurityPolicy)
 
 While the latter is more verbose, it also arguably makes it more obvious what's
 going on.  All of the :app:`Pyramid` core code uses this pattern rather than
diff --git a/docs/narr/advconfig.rst b/docs/narr/advconfig.rst
index 3ef350888..1d094f219 100644
--- a/docs/narr/advconfig.rst
+++ b/docs/narr/advconfig.rst
@@ -307,6 +307,7 @@ These are the methods of the configurator which provide conflict detection:
 :meth:`~pyramid.config.Configurator.set_view_mapper`,
 :meth:`~pyramid.config.Configurator.set_authentication_policy`,
 :meth:`~pyramid.config.Configurator.set_authorization_policy`,
+:meth:`~pyramid.config.Configurator.set_security_policy`,
 :meth:`~pyramid.config.Configurator.set_locale_negotiator`,
 :meth:`~pyramid.config.Configurator.set_default_permission`,
 :meth:`~pyramid.config.Configurator.add_traverser`,
diff --git a/docs/narr/extconfig.rst b/docs/narr/extconfig.rst
index 4c6c8b70b..5a99fc1c6 100644
--- a/docs/narr/extconfig.rst
+++ b/docs/narr/extconfig.rst
@@ -271,6 +271,7 @@ Pre-defined Phases
 
 - :meth:`pyramid.config.Configurator.add_route`
 - :meth:`pyramid.config.Configurator.set_authentication_policy`
+- :meth:`pyramid.config.Configurator.set_security_policy`
 
 :const:`pyramid.config.PHASE3_CONFIG`
 
diff --git a/docs/narr/introspector.rst b/docs/narr/introspector.rst
index 50f4ac736..40002347c 100644
--- a/docs/narr/introspector.rst
+++ b/docs/narr/introspector.rst
@@ -302,6 +302,16 @@ introspectables in categories not described here.
      The :class:`pyramid.interfaces.IRoute` object that is used to perform
      matching and generation for this route.
 
+``security policy``
+
+  There will be one and only one introspectable in the ``security policy`` category.
+  It represents a call to the :meth:`pyramid.config.Configurator.set_security_policy` method (or its Configurator constructor equivalent).
+  It will have the following data:
+
+  ``policy``
+
+    The policy object (the resolved ``policy`` argument to ``set_security_policy``).
+
 ``authentication policy``
 
   There will be one and only one introspectable in the ``authentication
diff --git a/docs/narr/testing.rst b/docs/narr/testing.rst
index 883bb7c7b..2182082a8 100644
--- a/docs/narr/testing.rst
+++ b/docs/narr/testing.rst
@@ -278,7 +278,7 @@ In the above example, we create a ``MyTest`` test case that inherits from
 be found when ``pytest`` is run.  It has two test methods.
 
 The first test method, ``test_view_fn_forbidden`` tests the ``view_fn`` when
-the authentication policy forbids the current user the ``edit`` permission. Its
+the security policy forbids the current user the ``edit`` permission. Its
 third line registers a "dummy" "non-permissive" authorization policy using the
 :meth:`~pyramid.config.Configurator.testing_securitypolicy` method, which is a
 special helper method for unit testing.
@@ -288,13 +288,13 @@ WebOb request object API.  A :class:`pyramid.testing.DummyRequest` is a request
 object that requires less setup than a "real" :app:`Pyramid` request.  We call
 the function being tested with the manufactured request.  When the function is
 called, :meth:`pyramid.request.Request.has_permission` will call the "dummy"
-authentication policy we've registered through
+security policy we've registered through
 :meth:`~pyramid.config.Configurator.testing_securitypolicy`, which denies
 access.  We check that the view function raises a
 :exc:`~pyramid.httpexceptions.HTTPForbidden` error.
 
 The second test method, named ``test_view_fn_allowed``, tests the alternate
-case, where the authentication policy allows access.  Notice that we pass
+case, where the security policy allows access.  Notice that we pass
 different values to :meth:`~pyramid.config.Configurator.testing_securitypolicy`
 to obtain this result.  We assert at the end of this that the view function
 returns a value.
diff --git a/docs/narr/threadlocals.rst b/docs/narr/threadlocals.rst
index 7437a3a76..8aa5b313d 100644
--- a/docs/narr/threadlocals.rst
+++ b/docs/narr/threadlocals.rst
@@ -32,11 +32,11 @@ various :app:`Pyramid` API functions.  For example, the implementation of the
 :mod:`pyramid.security` function named
 :func:`~pyramid.security.authenticated_userid` (deprecated as of 1.5) retrieves
 the thread local :term:`application registry` as a matter of course to find an
-:term:`authentication policy`.  It uses the
+:term:`security policy`.  It uses the
 :func:`pyramid.threadlocal.get_current_registry` function to retrieve the
-application registry, from which it looks up the authentication policy; it then
-uses the authentication policy to retrieve the authenticated user id.  This is
-how :app:`Pyramid` allows arbitrary authentication policies to be "plugged in".
+application registry, from which it looks up the security policy; it then
+uses the security policy to retrieve the authenticated user id.  This is
+how :app:`Pyramid` allows arbitrary security policies to be "plugged in".
 
 When they need to do so, :app:`Pyramid` internals use two API functions to
 retrieve the :term:`request` and :term:`application registry`:
diff --git a/src/pyramid/config/routes.py b/src/pyramid/config/routes.py
index daef8e9f2..ad846a107 100644
--- a/src/pyramid/config/routes.py
+++ b/src/pyramid/config/routes.py
@@ -335,9 +335,10 @@ class RoutesConfiguratorMixin(object):
         if 'effective_principals' in predicates:
             warnings.warn(
                 (
-                    'The new security policy has removed the concept of '
-                    'principals. See "Upgrading Authentication/Authorization" '
-                    'in "What\'s New in Pyramid 2.0" for more information.'
+                    'The new security policy has deprecated '
+                    'effective_principals. See "Upgrading '
+                    'Authentication/Authorization" in "What\'s New in '
+                    'Pyramid 2.0" for more information.'
                 ),
                 DeprecationWarning,
                 stacklevel=3,
diff --git a/src/pyramid/config/security.py b/src/pyramid/config/security.py
index 99eb5792c..8d6a417c0 100644
--- a/src/pyramid/config/security.py
+++ b/src/pyramid/config/security.py
@@ -1,5 +1,5 @@
+import warnings
 from zope.interface import implementer
-from zope.deprecation import deprecate
 
 from pyramid.interfaces import (
     IAuthorizationPolicy,
@@ -57,13 +57,6 @@ class SecurityConfiguratorMixin(object):
             introspectables=(intr,),
         )
 
-    @deprecate(
-        'Authentication and authorization policies have been deprecated in '
-        'favor of security policies.  See '
-        'https://docs.pylonsproject.org/projects/pyramid/en/latest'
-        '/whatsnew-2.0.html#upgrading-authentication-authorization '
-        'for more information.'
-    )
     @action_method
     def set_authentication_policy(self, policy):
         """
@@ -84,6 +77,14 @@ class SecurityConfiguratorMixin(object):
            achieve the same purpose.
 
         """
+        warnings.warn(
+            'Authentication and authorization policies have been deprecated '
+            'in favor of security policies.  See "Upgrading '
+            'Authentication/Authorization" in "What\'s New in Pyramid 2.0" '
+            'for more information.',
+            DeprecationWarning,
+            stacklevel=3,
+        )
 
         def register():
             self.registry.registerUtility(policy, IAuthenticationPolicy)
@@ -137,6 +138,14 @@ class SecurityConfiguratorMixin(object):
            achieve the same purpose.
 
         """
+        warnings.warn(
+            'Authentication and authorization policies have been deprecated '
+            'in favor of security policies.  See "Upgrading '
+            'Authentication/Authorization" in "What\'s New in Pyramid 2.0" '
+            'for more information.',
+            DeprecationWarning,
+            stacklevel=3,
+        )
 
         def register():
             self.registry.registerUtility(policy, IAuthorizationPolicy)
diff --git a/src/pyramid/config/views.py b/src/pyramid/config/views.py
index 324462d1a..2260d5d54 100644
--- a/src/pyramid/config/views.py
+++ b/src/pyramid/config/views.py
@@ -794,9 +794,10 @@ class ViewsConfiguratorMixin(object):
         if 'effective_principals' in view_options:
             warnings.warn(
                 (
-                    'The new security policy has removed the concept of '
-                    'principals. See "Upgrading Authentication/Authorization" '
-                    'in "What\'s New in Pyramid 2.0" for more information.'
+                    'The new security policy has deprecated '
+                    'effective_principals. See "Upgrading '
+                    'Authentication/Authorization" in "What\'s New in '
+                    'Pyramid 2.0" for more information.'
                 ),
                 DeprecationWarning,
                 stacklevel=4,
diff --git a/src/pyramid/security.py b/src/pyramid/security.py
index e3a978c52..2a0fb1279 100644
--- a/src/pyramid/security.py
+++ b/src/pyramid/security.py
@@ -41,10 +41,6 @@ def _get_security_policy(request):
     return request.registry.queryUtility(ISecurityPolicy)
 
 
-def _get_authentication_policy(request):
-    return request.registry.queryUtility(IAuthenticationPolicy)
-
-
 def remember(request, userid, **kw):
     """
     Returns a sequence of header tuples (e.g. ``[('Set-Cookie', 'foo=abc')]``)
@@ -71,7 +67,7 @@ def remember(request, userid, **kw):
 
     .. versionchanged:: 1.6
         Deprecated the ``principal`` argument in favor of ``userid`` to clarify
-        its relationship to the authentication policy.
+        its relationship to the security policy.
 
     .. versionchanged:: 1.10
         Removed the deprecated ``principal`` argument.
@@ -141,8 +137,7 @@ def principals_allowed_by_permission(context, permission):
 deprecated(
     'principals_allowed_by_permission',
     'The new security policy has removed the concept of principals.  See '
-    'https://docs.pylonsproject.org/projects/pyramid/en/latest'
-    '/whatsnew-2.0.html#upgrading-authentication-authorization '
+    '"Upgrading Authentication/Authorization" in "What\'s New in Pyramid 2.0" '
     'for more information.',
 )
 
@@ -152,7 +147,7 @@ def view_execution_permitted(context, request, name=''):
     by a :term:`permission`, check the permission associated with the
     view using the effective authentication/authorization policies and
     the ``request``.  Return a boolean result.  If no
-    :term:`authorization policy` is in effect, or if the view is not
+    :term:`security policy` is in effect, or if the view is not
     protected by a permission, return ``True``. If no view can view found,
     an exception will be raised.
 
@@ -376,14 +371,22 @@ class AuthenticationAPIMixin(object):
         associated with the userid exists in persistent storage.
 
         """
-        authn = _get_authentication_policy(self)
         security = _get_security_policy(self)
-        if authn is not None:
-            return authn.unauthenticated_userid(self)
-        elif security is not None:
-            return security.authenticated_userid(self)
-        else:
+        if security is None:
             return None
+        if isinstance(security, LegacySecurityPolicy):
+            authn = security._get_authn_policy(self)
+            return authn.unauthenticated_userid(self)
+        return security.authenticated_userid(self)
+
+    unauthenticated_userid = deprecated(
+        unauthenticated_userid,
+        (
+            'The new security policy has deprecated unauthenticated_userid. '
+            'See "Upgrading Authentication/Authorization" in "What\'s New in '
+            'Pyramid 2.0" for more information.'
+        ),
+    )
 
     @property
     def effective_principals(self):
@@ -399,17 +402,19 @@ class AuthenticationAPIMixin(object):
         :data:`pyramid.security.Everyone` principal.
 
         """
-        policy = _get_authentication_policy(self)
-        if policy is None:
-            return [Everyone]
-        return policy.effective_principals(self)
+        security = _get_security_policy(self)
+        if security is not None and isinstance(security, LegacySecurityPolicy):
+            authn = security._get_authn_policy(self)
+            return authn.effective_principals(self)
+        return [Everyone]
 
     effective_principals = deprecated(
         effective_principals,
-        'The new security policy has removed the concept of principals.  See '
-        'https://docs.pylonsproject.org/projects/pyramid/en/latest'
-        '/whatsnew-2.0.html#upgrading-authentication-authorization '
-        'for more information.',
+        (
+            'The new security policy has deprecated effective_principals. '
+            'See "Upgrading Authentication/Authorization" in "What\'s New in '
+            'Pyramid 2.0" for more information.'
+        ),
     )
 
 
diff --git a/src/pyramid/view.py b/src/pyramid/view.py
index eeac4e783..201e8af7c 100644
--- a/src/pyramid/view.py
+++ b/src/pyramid/view.py
@@ -102,7 +102,7 @@ def render_view_to_iterable(context, request, name='', secure=True):
     If ``secure`` is ``True``, and the view is protected by a permission, the
     permission will be checked before the view function is invoked.  If the
     permission check disallows view execution (based on the current
-    :term:`authentication policy`), a
+    :term:`security policy`), a
     :exc:`pyramid.httpexceptions.HTTPForbidden` exception will be raised; its
     ``args`` attribute explains why the view access was disallowed.
 
diff --git a/tests/test_config/test_routes.py b/tests/test_config/test_routes.py
index 423da5834..a75fdd776 100644
--- a/tests/test_config/test_routes.py
+++ b/tests/test_config/test_routes.py
@@ -316,7 +316,7 @@ class RoutesConfiguratorMixinTests(unittest.TestCase):
             warnings.simplefilter('always', DeprecationWarning)
             config.add_route('foo', '/bar', effective_principals=['any'])
             self.assertIn(
-                'removed the concept of principals', str(w[-1].message)
+                'deprecated effective_principals', str(w[-1].message)
             )
 
 
diff --git a/tests/test_config/test_views.py b/tests/test_config/test_views.py
index d133aedbd..353749ed6 100644
--- a/tests/test_config/test_views.py
+++ b/tests/test_config/test_views.py
@@ -2933,7 +2933,7 @@ class TestViewsConfigurationMixin(unittest.TestCase):
             warnings.simplefilter('always', DeprecationWarning)
             config.add_view(lambda: None, effective_principals=['any'])
             self.assertIn(
-                'removed the concept of principals', str(w[-1].message)
+                'deprecated effective_principals', str(w[-1].message)
             )
 
 
diff --git a/tests/test_predicates.py b/tests/test_predicates.py
index 4029faf9d..c0a6c59ec 100644
--- a/tests/test_predicates.py
+++ b/tests/test_predicates.py
@@ -454,8 +454,9 @@ class Test_EffectivePrincipalsPredicate(unittest.TestCase):
         return EffectivePrincipalsPredicate(val, config)
 
     def _testing_authn_policy(self, userid, groupids=tuple()):
-        from pyramid.interfaces import IAuthenticationPolicy
+        from pyramid.interfaces import IAuthenticationPolicy, ISecurityPolicy
         from pyramid.security import Everyone, Authenticated
+        from pyramid.security import LegacySecurityPolicy
 
         class DummyPolicy:
             def effective_principals(self, request):
@@ -468,6 +469,7 @@ class Test_EffectivePrincipalsPredicate(unittest.TestCase):
 
         registry = self.config.registry
         registry.registerUtility(DummyPolicy(), IAuthenticationPolicy)
+        registry.registerUtility(LegacySecurityPolicy(), ISecurityPolicy)
 
     def test_text(self):
         inst = self._makeOne(('verna', 'fred'), None)
diff --git a/tests/test_router.py b/tests/test_router.py
index f6b7f64d3..6fa9f9a5b 100644
--- a/tests/test_router.py
+++ b/tests/test_router.py
@@ -1699,10 +1699,6 @@ class DummyResponse(object):
         return self.app_iter
 
 
-class DummyAuthenticationPolicy:
-    pass
-
-
 class DummyLogger:
     def __init__(self):
         self.messages = []
diff --git a/tests/test_security.py b/tests/test_security.py
index f39e3c730..fa3d165ea 100644
--- a/tests/test_security.py
+++ b/tests/test_security.py
@@ -346,16 +346,22 @@ class TestAuthenticatedUserId(unittest.TestCase):
         request = _makeRequest()
         self.assertEqual(request.authenticated_userid, None)
 
+    def test_with_security_policy(self):
+        request = _makeRequest()
+        _registerSecurityPolicy(request.registry, '123')
+        self.assertEqual(request.authenticated_userid, '123')
+
     def test_with_authentication_policy(self):
         request = _makeRequest()
         _registerAuthenticationPolicy(request.registry, 'yo')
-        _registerSecurityPolicy(request.registry, 'wat')
-        self.assertEqual(request.authenticated_userid, 'wat')
+        _registerLegacySecurityPolicy(request.registry)
+        self.assertEqual(request.authenticated_userid, 'yo')
 
-    def test_with_security_policy(self):
+    def test_security_policy_trumps_authentication_policy(self):
         request = _makeRequest()
-        _registerSecurityPolicy(request.registry, '123')
-        self.assertEqual(request.authenticated_userid, '123')
+        _registerAuthenticationPolicy(request.registry, 'yo')
+        _registerSecurityPolicy(request.registry, 'wat')
+        self.assertEqual(request.authenticated_userid, 'wat')
 
 
 class TestUnAuthenticatedUserId(unittest.TestCase):
@@ -369,17 +375,23 @@ class TestUnAuthenticatedUserId(unittest.TestCase):
         request = _makeRequest()
         self.assertEqual(request.unauthenticated_userid, None)
 
-    def test_with_authentication_policy(self):
+    def test_with_security_policy(self):
         request = _makeRequest()
-        _registerAuthenticationPolicy(request.registry, 'yo')
-        _registerSecurityPolicy(request.registry, 'wat')
+        _registerSecurityPolicy(request.registry, 'yo')
         self.assertEqual(request.unauthenticated_userid, 'yo')
 
-    def test_with_security_policy(self):
+    def test_legacy_authentication_policy(self):
         request = _makeRequest()
-        _registerSecurityPolicy(request.registry, 'yo')
+        _registerAuthenticationPolicy(request.registry, 'yo')
+        _registerLegacySecurityPolicy(request.registry)
         self.assertEqual(request.unauthenticated_userid, 'yo')
 
+    def test_security_policy_trumps_authentication_policy(self):
+        request = _makeRequest()
+        _registerAuthenticationPolicy(request.registry, 'yo')
+        _registerSecurityPolicy(request.registry, 'wat')
+        self.assertEqual(request.unauthenticated_userid, 'wat')
+
 
 class TestEffectivePrincipals(unittest.TestCase):
     def setUp(self):
@@ -394,11 +406,27 @@ class TestEffectivePrincipals(unittest.TestCase):
         request = _makeRequest()
         self.assertEqual(request.effective_principals, [Everyone])
 
-    def test_with_authentication_policy(self):
+    def test_with_security_policy(self):
+        from pyramid.security import Everyone
+
+        request = _makeRequest()
+        _registerSecurityPolicy(request.registry, 'yo')
+        self.assertEqual(request.effective_principals, [Everyone])
+
+    def test_legacy_authentication_policy(self):
         request = _makeRequest()
         _registerAuthenticationPolicy(request.registry, 'yo')
+        _registerLegacySecurityPolicy(request.registry)
         self.assertEqual(request.effective_principals, 'yo')
 
+    def test_security_policy_trumps_authentication_policy(self):
+        from pyramid.security import Everyone
+
+        request = _makeRequest()
+        _registerAuthenticationPolicy(request.registry, 'wat')
+        _registerSecurityPolicy(request.registry, 'yo')
+        self.assertEqual(request.effective_principals, [Everyone])
+
 
 class TestHasPermission(unittest.TestCase):
     def setUp(self):
@@ -567,6 +595,15 @@ def _registerSecurityPolicy(reg, result):
     return policy
 
 
+def _registerLegacySecurityPolicy(reg):
+    from pyramid.interfaces import ISecurityPolicy
+    from pyramid.security import LegacySecurityPolicy
+
+    policy = LegacySecurityPolicy()
+    reg.registerUtility(policy, ISecurityPolicy)
+    return policy
+
+
 def _registerAuthenticationPolicy(reg, result):
     from pyramid.interfaces import IAuthenticationPolicy