Defending BFG's Design ====================== From time to time, challenges to various aspects of :mod:`repoze.bfg` design are lodged. To give context to the discussions that follow, we detail some of the design decisions and tradeoffs here. BFG Uses The Zope Component Architecture ("ZCA") ------------------------------------------------ BFG uses the :term:`Zope Component Architecture` (ZCA) "under the hood". This is a point of some contention. :mod:`repoze.bfg` is of a :term:`Zope` pedigree, so it was natural for it developers to use the ZCA at its inception. However, :mod:`repoze.bfg` allegiance to its Zope pedigree is not blind. We understand that using the ZCA has issues and consequences, which we've attempted to address as best we can. Here'a a full disclosure about BFG's use of the ZCA, and the tradeoffs its usage involves. Problems ++++++++ The API that is commonly used to access data in a ZCA "component registry" is not particularly pretty or intuitive, and sometimes it's just plain obtuse. Likewise, the conceptual load on a casual source code reader of code that uses the component architecture is somewhat high. Consider a ZCA neophyte reading the code that performs a typical "unnamed utility" lookup: .. code-block:: python :linenos: from repoze.bfg.interfaces import ISettings from zope.component import getUtility settings = getUtility(ISettings) After this code runs, ``settings`` will be a Python dictionary. But it's unlikely that any "civilian" would know that just by reading the code. There are a number of comprehension issues with the bit of code above that are pretty obvious. First, what's a "utility"? Well, for the purposes of this dicussion, and for the purpose of the code above, it's just not very important. If you really care, you can go read `this `. However, still folks need to understand the concept in order to parse the code while reading it. This is problem number one. Second, what's this ``ISettings`` thing? Well, it's an :term:`interface`. Is that important here? Not really, we're just using it as a "key" for some lookup based on its identity as a marker: it represents an object that has the dictionary API, but that's not really very important. That's problem number two. Third of all, what does ``getUtility`` do? It's performing a lookup for the ``ISettings`` "utility" that should return.. well, a utility. Note how we've already built up a dependency on the understanding of an :term:`interface` and the concept of "utility" to answer this question: a bad sign so far. Note also that the answer is circular, a *really* bad sign. Fourth, where does ``getUtility`` look to get the data? Well, the "component registry" of course. What's component registry? Right. Problem number four. Fifth, assuming you buy that there's some magical registry hanging around, where *is* this registry? Homina homina... "around"? That's sort of the best answer in this context (a more specific answer would require knowledge of internals). Can there be more than one? Yes. So *which* registry does it find the registration in? Well, the "current" registry of course. In terms of :mod:`repoze.bfg`, the current registry is a thread local variable. Using an API that consults a thread local makes understanding how it works nonlocal. Sixth, fine, you've bought in to the fact that there's a registry that is just "hanging around". But how does the registry get populated? Why, :term:`ZCML` of course. Sometimes. In this particular case, however, the registration of ``ISettings`` is made by the framework itself "under the hood": it's not present in any ZCML. This is extremely hard to comprehend. Clearly there's some amount of cognitive load here that needs to be borne by a reader of code that extends the :mod:`repoze.bfg` framework due to its use of the ZCA, even if he or she is already an expert Python programmer and whom is an expert in the domain of web applications. This is suboptimal. Ameliorations +++++++++++++ First, the biggest amelioration: :mod:`repoze.bfg` *does not expect application developers to understand ZCA concepts or its API*. If an *application* developer needs to understand a ZCA concept or API during the creation of a :mod:`repoze.bfg` application, we've failed on some axis. Instead, the framework hides the presence of the ZCA behind special-purpose API functions that *do* use the ZCA API. Take for example the ``repoze.bfg.security.authenticated_userid`` function, which returns the userid present in the current request or ``None`` if no userid is present in the current request. The application developer calls it like so: .. code-block:: python :linenos: from repoze.bfg.security import authenticated_userid userid = authenticated_userid(request) He now has the current user id. Under its hood however, the implementation of ``authenticated_userid`` is this: .. code-block: python :linenos: def authenticated_userid(request): """ Return the userid of the currently authenticated user or ``None`` if there is no authentication policy in effect or there is no currently authenticated user. """ policy = queryUtility(IAuthenticationPolicy) if policy is None: return None return policy.authenticated_userid(request) Using such wrappers, we strive to always hide the ZCA this way from application developers: application developers should just never know about the ZCA. They call a function with some object germane to the domain, it returns a result; they needn't understand components at all. A corollary that follows is that any reader of an application that has been written using :mod:`repoze.bfg` needn't understand the ZCA either. Hiding the ZCA from application developers and code readers in this way a form of enhancing "domain specificity". No end user wants to understand the minutiae of the mechanics of how a web framework does its thing. People want to deal in concepts that are closer to the domain they're working in: for example, web developers want to know about *users*, not *utilities*. :mod:`repoze.bfg` uses the ZCA as an implementation detail, not as a feature which is exposed to end users. However, unlike application developers, BFG *framework developers*, including people who want to override :mod:`repoze.bfg` functionality via preordained framework plugpoints like traversal or view lookup *must* understand the ZCA. :mod:`repoze.bfg` framework developers were so concerned about conceptual load issues of the ZCA API for framework developers that a `replacement `_ was actually developed. Though this package is fully functional and well-tested, and its API is much nicer than the ZCA API, work on it was largely abandoned and it is not used in :mod:`repoze.bfg`. We continued to use the ZCA within :mod:`repoze.bfg`. Making framework developers and extenders understand the ZCA is a tradeoff. We (the :mod:`repoze.bfg` developers) like the features that the ZCA gives us, and we have long-ago borne the weight of understanding what it does and how it works. The authors of :mod:`repoze.bfg` understand the ZCA deeply and can read code that uses it as easily as any other code. However, we do recognize that other developers who my want to extend the framework are not as comfortable with ZCA we are with it. So, for the purposes of being kind to framework developers who may be dismayed by some of the more flagrant uses of the ZCA API in :mod:`repoze.bfg`, we've turned the component registry used in BFG into something that is accessible using the plain old dictionary API (like the :mod:`repoze.component` API). Our example in the problem section above was: .. code-block:: python :linenos: from repoze.bfg.interfaces import ISettings from zope.component import getUtility settings = getUtility(ISettings) In a better world, we might be able to spell this as: .. code-block:: python :linenos: from repoze.bfg.threadlocal import get_registry registry = get_registry() settings = registry['settings'] In this world, we've removed the need to understand utilities and interfaces. We *haven't* removed the need to understand the concept of a *registry*, but for the purposes of this example, it's simply a dictionary. We haven't killed off the concept of a thread local either. Let's kill off thread locals, pretending to want to do this in some code that has access to the :term:`request`: .. code-block:: python :linenos: registry = request.registry settings = registry['settings'] In *this* world, we've reduced the conceptual problem to understanding attributes and the dictionary API. Every Python programmer knows these things, even framework programmers! We continued using ZCA rather than ditching it for :mod:`repoze.component` largely because the ZCA concept of interfaces provides for use of an interface hierarchy, which is useful in a lot of scenarios (such as context type inheritance). Coming up with a marker type that was something like an interface that allowed for this functionality seemed like it was just reinventing the wheel. While :mod:`repoze.bfg` still uses some suboptimal unnamed utility registrations, future versions of it will (where possible) disuse these things in favor of straight dictionary assignments and lookups, as demonstrated above, to be kinder to new developers and extenders. We'll continue to seek ways to reduce framework extender cognitive load. Rationale +++++++++ Here are the main rationales for BFG's design decision to use the ZCA: - Pedigree. A nontrivial part of the answer to this question is "pedigree". Much of the design of :mod:`repoze.bfg` is stolen directly from :term:`Zope`. Zope uses the ZCA to do a number of tricks. :mod:`repoze.bfg` mimics these tricks apeishly, and, because the ZCA works pretty well for that set of tricks, the :mod:`repoze.bfg` mimicry uses it for the same purposes. For example, the way that BFG maps a :term:`request` to a :term:`view callable` is lifted almost entirely from Zope. The ZCA plays an important role in the particulars of how this request to view mapping is done. - Features. The ZCA essentially provides what can be considered something like a "superdictionary", which allows for more complex lookups than retrieving a value based on a single key. Some of this lookup capability is very useful for end users, such as being able to register a view that is only found when the context is some class of object, or when the context implements some :term:`interface`. - Singularity. There's only one "place" where "application configuration" lives in a BFG application: in a component registry. The component registry answers questions made to it by the framework at runtime based on the configuration of *an application*. Note: "an application" is not the same as "a process", multiple independently configured copies of the same BFG application are capable of running in the same process space. - Composability. A ZCA registry can be populated imperatively, or there's an existing mechanism to populate a registry via the use of a configuration file (ZCML). We didn't need to write a frontend from scratch to make use of configuration-file-driven registry population. - Pluggability. Use of the ZCA allows for framework extensibility via a well-defined and widely understood plugin architecture. As long as framework developers and extenders understand the ZCA, it's possible to extend BFG almost arbitrarily. For example, it's relatively easy to build a ZCML directive that registers several views "all at once", allowing app developers to use that ZCML directive as a "macro" in code that they write. This is somewhat of a differentiating feature from other (non-Zope) frameworks. - Testability. Judicious use of the ZCA in framework code makes testing that code slightly easier. Instead of using monkeypatching or other facilities to register mock objects for testing, we inject dependencies via ZCA registrations and then use lookups in the code find our mock objects. - Speed. The ZCA is very fast for a specific set of complex lookup scenarios that BFG uses, having been optimized through the years for just these purposes. The ZCA contains optional C code for this purpose which demonstrably has no (or very few) bugs. - Ecosystem. Many existing Zope packages can be used in :mod:`repoze.bfg` with few (or no) changes due to our use of the ZCA and :term:`ZCML`. Conclusion ++++++++++ If you only *develop applications* using :mod:`repoze.bfg`, there's just basically nothing to think about here. You just should never need to understand the ZCA or even know about its presence: use documented APIs instead. If you're an application developer who doesn't read API documentation because its unmanly, but instead uses raw source code, and considers everything an API, and you've pained yourself into a conceptual corner as a result of needing to wrestle with some ZCA-using internals, it's hard to have a lot of sympathy for you. You'll either need to get familiar with how we're using the ZCA or you'll need to use only the documented APIs; that's why we document 'em. If you *extend* or *develop* :mod:`repoze.bfg` (create new ZCML directives, use some of the more obscure "ZCML hooks" as described in :ref:`hooks_chapter`, or work on the :mod:`repoze.bfg` core code), you will be faced with needing to understand at least some ZCA concepts. The ZCA API is pretty quirky: we've tried to make it at least slightly nicer by disusing it for common registrations and lookups such as unnamed utilities. Some places it's used unabashedly, and will be forever. We know it's a bit quirky, but it's also useful and fundamentally understandable if you take the time to do some reading about it. BFG "Encourages Use of ZCML" ---------------------------- :term:`ZCML` is a configuration language that can be used to configure the :term:`Zope Component Architecture` registry that BFG uses as its application configuration. Quick answer: well, no, it doesn't.. not really. You can use the ``bfg_view`` decorator for the most common form of configuration. But, yes, your application currently does need to possess a ZCML file for it to begin executing successfully even if its only contents are a ```` directive that kicks off the location of decorated views. In any case, in the interest of completeness and in the spirit of , BFG 1.2 will include a completely imperative mode for all configuration. You will be able to make "single file" apps in this mode, which should help people who need to see everything done completely imperatively. For example, the very most basic :mod:`repoze.bfg` "helloworld" program will become something like:: from webob import Response from wsgiref import simple_server from repoze.bfg.registry import Registry from repoze.bfg.router import Router def helloworld_view(request): return Response(hello') if __name__ == '__main__': reg = Registry() reg.view(helloworld_view) app = Router(reg) simple_server.make_server('', 8080, app).serve_forever() In this mode, no ZCML will be required. Hopefully this mode will allow people who are used to doing everything imperatively feel more comfortable. Other Topics ------------ We'll be trying to cover the following in this document as time allows: - BFG View Lookup and Registration Is "Complex" - BFG Template Lookup Is "Complex" - BFG Views Do Not Accept Arbitrary Keyword Arguments - BFG Does Traversal, And I Don't Like Traversal - BFG Does URL Dispatch, And I Don't Like URL Dispatch Other challenges are encouraged to be sent to the `Repoze-Dev `_ maillist. We'll try to address them by considering a design change, or at very least via exposition here.