[Zope3-dev] Zope 3 releases?

Hi Jim

An: zope3-dev Development
Betreff: [Zope3-dev] Zope 3 releases?

[...]

- We need to decide what a Zope 3 release is (or maybe
multiple flavors). I favor copying the linux experiences,
but have an open mind.

I think there are two different point of views. Zope as a library
and Zope as a product.

I think you are asking for Zope 3 as a component library.
As a developer I'm happy with this kind of library and don't
need a Zope 3 application server release.

But I also see another point of view. Zope 3 as a product we
can lobby for and a application server which is ready to use
with a easy setup. e.g. windows installer or buildout,
easy install.

I think such a Zope 3 application server has the following
benefit.

- easy to install and ready to use for newbees or small projects
- a proof that we are able to setup such a working server
- a working setup which 3rd party developer can develop with
- a product which the sales can show and lobby for

But who will do this?
Probably we need a own community which picks up this
task and supports a Zope application server built
with cool zope components out there.

I guess we will see some different Releases in the near
future based on zope components e.g.

- Grok
- Z3C
- Plone 3.0

What do you think is somebody interested in doing
another release based on zope components?

And if so, does this make a Zope 3 release obsolate?

Regards
Roger Ineichen

Jim
--
Jim Fulton
Zope Corporation
_______________________________________________
Zope3-dev mailing list
http://mail.zope.org/mailman/options/zope3-dev/dev%40projekt01.ch

Martijn Faassen

2007-10-06 23:22:25 UTC

Roger Ineichen wrote:
[snip]

Post by Roger Ineichen
But I also see another point of view. Zope 3 as a product we
can lobby for and a application server which is ready to use
with a easy setup. e.g. windows installer or buildout,
easy install.
I think such a Zope 3 application server has the following
benefit.
- easy to install and ready to use for newbees or small projects
- a proof that we are able to setup such a working server
- a working setup which 3rd party developer can develop with
- a product which the sales can show and lobby for
But who will do this?

I know who, I know who! :)

Post by Roger Ineichen
Probably we need a own community which picks up this
task and supports a Zope application server built
with cool zope components out there.

And we have such a community: Grok!

See grok.zope.org and grok-dev for the community that is doing what you
describe for the last year or so.

This is *exactly* what is Grok is about. I'm not kidding. We don't call
it Zope 3. We call it "Grok" or "Zope Grok". We build on the giant that
is Zope 3. This way we avoid the confusion on whether Zope 3 is a set of
libraries or a web framework you can install and try out. Grok's the latter.

I can imagine you don't agree with some of the choices we made with
Grok. I'd be happy to see other communities that try to do the same
thing. It'll only benefit the underlying technology of Grok, after all.

One outcome of this discussion might be that we all indeed agree with
the naming choice we made with Grok: we don't want to call such a web
application framework "Zope 3", but something else, to separate the
concerns.

Regards,

Martijn

Stephan Richter

2007-10-07 01:53:10 UTC

Post by Jim Fulton
I have some ideas and observations. I don't have decisions at this
point. We need to make some decisions as a community, with me
hopefully helping at times to get us unstuck. :)

I agree.

Post by Jim Fulton
- The classic 3.4 release seems to be stalled.

This is actually very sad. Let's take a stab back. The current story that we
have for the outside world is still the same as for Zope 3.3, simply because
we have not released anything else officially yet. That means, people have
the expectation of a Zope 3.4 tar ball.

I think there are two possible solutions to this:

(I) Keep the status quo for one more release.

While the 3.4 release effort has stalled, it is not dead. However, I would
take a different approach this time.

1. Create final 3.4.x releases of all remaining packages. A list of packages
that still have to be packaged can be found here:

http://wiki.zope.org/zope3/StabilizeEggPackages

2. With the new Zope 3.4 index in place, we can now test the stable set; see
my other post how to do that. The index is at:

http://download.zope.org/zope3.4/

3. Once the Zope 3.4 index is stable, a small tool can be used to build a
classic Zope 3.4 tar ball release.

The nice part about this approach is that it requires very little overhead to
what we have to do anyways.

(II) Create a KGS and document the new way

We create a Zope 3.4 index as pointed out above, but instead of releasing a
big tar ball, we write documentation on how to get started with the egg
index. The documentation should have topics like:

- How to get started with buildout.
- How to convert a classic Zope 3 projects to using eggs.
- Using Zope 3.4 eggs without writing your own.

Over the years I have become weary of hoping for people to write
documentation. While I think that we need this sort of documentation badly
and latest for Zope 3.5, I doubt it will happen for 3.4. I would love to be
prooven wrong!!

Post by Jim Fulton
- There is a desperate need for a known good configuration of
components that work well together and that people can build on
without having to think too hard about the underlying platform.

That's the obvious one. I really hope that the other discussion on KGSs will
gain enough support and a small, smooth workflow that acceptable for
everyone.

Post by Jim Fulton
- There is a need for a mechanism for testing "core" components to
make sure they don't cause breakage.

Yes, further we learned that the stitched trunk does not serve this need well
enough anymore. Luckily the KGS script I wrote to build an overall
buildout.cfg from the new Zope 3.4 index provides a good testing environment.
(I'll note that several issues with the released packages already surfaced.)

Post by Jim Fulton
- We need to decide what Zope 3 is.

I think this is much more difficult.

- I agree, Zope 3 is *not* an application.

- Also, for me, Zope 3 is *not just* a set of components or libraries.

- I like the suggestions of "platform and "environment".

More recently, I was asked to make the case for Zope 3 in very conservative
businesses. I have found that Zope 3 cannot be compared to PHP, RoR and so
on, but competes much more on the level of Java Application Servers, such as
BEA Weblogic and IBM Websphere. In fact, the Java solutions are pretty
similar in to what we have to offer. While we do not have the marketing
behind Zope 3 as the Java guys have, it should not stop us to call Zope 3
an "application server". So that's my entry.

Post by Jim Fulton
I think it's time we came up
with the elevator speech for what Zope 3 is. It should be a few
sentences at most.

Okay, instead of forming sentences, let me try to come up with a few concepts
that I think are central:

- complete stack of libraries geared towards Web applications
- enterprise-grade[1] technology
- built from experience
- written in Python
- compliant with many standards
- fully customizable due to CA

.. [1] under this buzz-word I mean we provide all the features critically in a
large-scale deployment, such as transaction-safety, scalability, stability
(tests), etc.

So here is an attempt:

Zope 3 is an enterprise-grade application server written in the Python
programming language. Built on many years of experience, Zope 3's components
implement many technical standards . Thanks to its component architecture, it
empowers the developer to provide custom implementations as desired. Zope 3
is mainly geared towards building Web applications, but can also be used to
built other applications.

Okay, okay, this might sound too much like marketing talk; maybe we should be
more technical?

Post by Jim Fulton
It need not be carved in stone forever, but it
should be agreed on and used for tactical planning. This should
probably go hand in hand with the bigger definition of what Zope is
along the lines of the ideas that Tres expressed at the DZUG in Potsdam.

What are Tres' ideas? I have not been at DZUG, so I have no idea.

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I like the Linux parallel as well. I think it would be nice, if we treat the
Zope 3 name like Debian, and Grok or other frameworks on top of it like
Knoppix or other Debian-derived distributions.

Post by Jim Fulton
- We need a *realistic* (especially wrt available resources) process
for managing releases.

Yes. I think one of the fatal mistakes of the eggification process was to
over-estimate the resources available in the Zope community.

I think with KGS we have a means to set realistic goals. Once we have an
initial stable set of releases[2], we can maintain the Zope 3.4 index as long
as we want to. In the mean time, we will have an unstable index where we can
test new feature releases. Arriving at a new stable index will then be a
matter of (a) creating final releases for all packages in the index and (b)
ensuring that all tests pass.

.. [2] We are not that far away from that. We currently have 32 failures and 6
errors out of 10.3k tests in the Zope 3.4 index.

Regards,
Stephan

--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training

Brandon Craig Rhodes

2007-10-07 07:20:14 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I like the Linux parallel as well. I think it would be nice, if we
treat the Zope 3 name like Debian, and Grok or other frameworks on
top of it like Knoppix or other Debian-derived distributions.

There is danger here. Many in the industry consider Linux to be an
absolute disaster - because after you develop an application on one
distribution, you can spend months trying to support customers who
attempt to run it on other distributions! In the wider Unix world
things are even worse. Tools like "autoconf" absorb months and years
of developer effort to simply get products to compile everywhere.

I will suggest an alternative model: Python.

The Zope project should not operate like Linux, but like Python.

The magic of Python is that it comes with a Standard Library. I am
sure, of course, that it takes the Python developers much more time to
come out with each version of Python because they have to keep the
Standard Library working; I am sure it is troublesome trying to get
all the volunteer maintainers on board when they want the next release
to meet its deadline; and it seems obvious that both Python and each
Standard Library module could evolve faster on their own, without
their releases tied to each other.

But what an immense benefit results! The normal Python coder, who
stands outside the development of the language itself, needs to only
be told "this is Python 2.4", and knows automatically not only which
core features the language will have, but, also, what versions of one
hundred other packages he will be working with. Hearing "2.5" not
only tells you that iterator comprehensions are available, but that
"datetime" has finally grown a .strptime class method!

Now, it sounds like Zope is about to stop being like Python and start
being like the current Wild West of Linux distributions. If I want to
write a WebWidget in the future, and want it to work with Grok, and
Zope on Wheels, and ZopeSprockets, and whatever other frameworks might
come along, then I will be faced with situations like "Well, Grok has
already moved up to zope.security 3.7, which means I can use key cards
natively! Hmm, but the other frameworks have not upgraded past 3.6; I
will have to special-case key-cards for them. On the other hand, Grok
stayed with zope.templates 3.5 because they piled so many extensions
on top that they didn't get time to redevelop them for 3.6, so I'm
going to have to fake push-masking in Grok..."

In conclusion (and there's a dozen more arguments I want to make, but
I think they're all pretty obvious if you start drawing more analogies
like the ones above), I think that breaking Zope into eggs makes about
as much sense as shipping Python bare and making users assemble
working sets of the Standard Library out of Cheeseshop eggs. While
perhaps saving a bit of time for the core developers, and perhaps
letting new releases come a bit faster since they are not all linked
together in a big release, the result of such a move will be to
inflict a terrible cost on the normal developer, who will have to
re-invent "autoconf" for Zope in order to have any hope of people
using his WebWidget.

While I'm being all contrary, I might as well make a whimsical
suggestion, a small gedankenexperiment: what if we went beyond the
suggestion that I am making here - which is that all "core" Zope 3
eggs continue to be released at the same time with a "big version
number" attached, exactly like Python and its Standard Library are -
and actually encouraged other tools and frameworks to "join in" and
follow the same cycle? What, in other words, if there were some way
for the WebWidget author - who, you will recall, wants people using
all sorts of Zope-based frameworks to be able to use his product - to
package his product such that you got the "right version" for the
version of Zope you were using? So that someone who put "webwidget"
in their "install_requires" stanza would get the "3.4" version if they
were using "Zope 3.4", and "3.5" if their Zope version was "3.5", and
so forth?

Then, not only would all of the Zope core packages be evolving in
lockstep, but third-party tools could voluntarily "follow along" with
the release cycle schedule, and be able to guarantee that things kept
working smoothly for their users.

It would be like having to grab the Python-LDAP that claims to work
with the Python version you're working with, but where the selection
was made automatically without your having to think about it.

--
Brandon Craig Rhodes ***@rhodesmill.org http://rhodesmill.org/brandon

Lennart Regebro

2007-10-07 12:16:40 UTC

Post by Brandon Craig Rhodes
There is danger here. Many in the industry consider Linux to be an
absolute disaster - because after you develop an application on one
distribution, you can spend months trying to support customers who
attempt to run it on other distributions! In the wider Unix world
things are even worse. Tools like "autoconf" absorb months and years
of developer effort to simply get products to compile everywhere.

Right, but the problem I think is that people say that applications
run on Linux, and also spend time to make them do so, and that needs a
lot of time spent.

I think that an application running on a pure Zope 3 would run also on
Grok, but it would be entirely possible to make another application
that for example doesn't use the ZODB at all, but still uses the rest
of the Zope3 "ecosystem". And an application running on Zope 3.3,
would not run on such a distribution.

So if we make Zope3 less of an application and more of a
platform/environment/ecosystem, we should probably not say that
anything runs "on" Zope3, and that would then solve that problem.
Applications would *run on* Grok, but *use* zope 3.

But I suspect that we then end up with your "Python" model, right? :-)

Post by Brandon Craig Rhodes
Now, it sounds like Zope is about to stop being like Python and start
being like the current Wild West of Linux distributions. If I want to
write a WebWidget in the future, and want it to work with Grok, and
Zope on Wheels, and ZopeSprockets, and whatever other frameworks might
come along, then I will be faced with situations like "Well, Grok has
already moved up to zope.security 3.7, which means I can use key cards
natively! Hmm, but the other frameworks have not upgraded past 3.6; I
will have to special-case key-cards for them. On the other hand, Grok
stayed with zope.templates 3.5 because they piled so many extensions
on top that they didn't get time to redevelop them for 3.6, so I'm
going to have to fake push-masking in Grok..."

Yeeeessss, I see your point. But is this avoidable at all? Even with
more fixed sets of versions we have this exact problem already today.

"I would like to use utilities, but Plone 2.5 is still on Zope 2.9
which has a slightly different utility implementation than Plone 3, so
I'm going to have to specialcase for that, and at the same time I'd
like to use this under Grok which has moved on to zope 3.4"... and so
on. I think that with the granular versions we get with eggs, it's in
fact gonna be *easier* to avoid this stuff, because I could in my
application update to a newer version of the module I need, and it
wouldn't actually break anything except in those cases where the
module breaks backwards compatibility, which isn't supposed to happen,
except in some extreme circumstances nowadays.

--
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

Jim Fulton

2007-10-07 17:06:54 UTC

IMO, the Python standard library and "batteries included" is a mess.
Despite being a Python contributor, I've very unmotivated to be a
contributor because the time lag between contributing and deriving
benefit from the contributions is too long. People had similar
complaints about Zope releases. I'll note that the problem is
exacerbated by the incompatibilities that (to some degree inevitably)
often occur in Python releases. Big-bang releases don't prevent
update problems.

Jim

--
Jim Fulton
Zope Corporation

Martijn Faassen

2007-10-07 22:09:25 UTC

Post by Jim Fulton
IMO, the Python standard library and "batteries included" is a mess.
Despite being a Python contributor, I've very unmotivated to be a
contributor because the time lag between contributing and deriving
benefit from the contributions is too long. People had similar
complaints about Zope releases. I'll note that the problem is
exacerbated by the incompatibilities that (to some degree inevitably)
often occur in Python releases. Big-bang releases don't prevent update
problems.

While I see Brandon making good points, I also agree with this: the
motivation to contribute to the Python standard library is reduced
because of big bang releases. I think the main reason is that you need
to wait so long to see your work "in print" so you can use it yourself.
(the other one is particular to Python core developer culture, who I
believe are more interested in language tweaks than library improvements).

Instead if I just release my own library I can release and use it
immediately (and get others to use it). I've seen the same with Zope 3
releases in the past. I think a predictable, regular release cycle can
help there, even though that has its own problems.

Splitting Zope up into independently evolving parts can help too. It
also has its own problems, some of which Brandon just pointed out. I've
also see the benefit of this already: Philipp can work on WSGI support
in zope.app.publisher or whatever and get it released so that, say,
Grok, could start using it, without having to wait for a major Zope 3
release, which we've historically not been very good at producing
regularly anyway.

It also means that one or more other projects will need to take over the
task of producing "big bang" releases out of a collection of these
smaller releases. This could be the index we've been speaking about, or
Zope 2, or Grok, or all of them combined. I will agree that Zope 3 is
somewhat special in that it tends to be more of an ecosystem than a
piece of software.

Regards,

Martijn

Lennart Regebro

2007-10-07 23:56:15 UTC

Post by Martijn Faassen
While I see Brandon making good points, I also agree with this: the
motivation to contribute to the Python standard library is reduced
because of big bang releases. I think the main reason is that you need
to wait so long to see your work "in print" so you can use it yourself.

Yup. This may be one of the reasons behind the lack of "kicking
upwards", as I outlined here:
http://blogs.nuxeo.com/sections/blogs/lennart_regebro/2005_04_11_sickness_zope

--
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

Lennart Regebro

2007-10-07 11:25:52 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I'm not sure what you mean with that, but for me, a known good set, or
what you call it, plus a buildout that uses this, would be enough for
a Zope 3.4 release.
If we easily can build the old standard downloadable tgz as well, then
fine, but it's not necessary.

Post by Jim Fulton
- We need a *realistic* (especially wrt available resources) process
for managing releases. There are 2 aspects of this. We shouldn't
make plans for which there aren't enough resources. We also
shouldn't plan significant tasks that people won't care enough to
work on. I think the classic Zope 3.4 release is a good example of a
large effort that really wouldn't benefit many people, if any.

Do you have a sort explanation on what is the missing resource? Is it,
as it was for 3.3, lack of people-hours with knowledge in fixing the
last bugs?

--
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

Jim Fulton

2007-10-07 17:09:39 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I'm not sure what you mean with that,

I mean we need to decide what a release is.

Post by Lennart Regebro
but for me, a known good set, or
what you call it, plus a buildout that uses this, would be enough for
a Zope 3.4 release.
If we easily can build the old standard downloadable tgz as well, then
fine, but it's not necessary.

That is my opinion too.

Do you have a sort explanation on what is the missing resource? Is it,
as it was for 3.3, lack of people-hours with knowledge in fixing the
last bugs?

I'm not entirely sure. I just observe that this doesn't seem to be
making much progress.

Jim

--
Jim Fulton
Zope Corporation

Martijn Faassen

2007-10-07 22:14:11 UTC

[snip]

Do you have a sort explanation on what is the missing resource? Is it,
as it was for 3.3, lack of people-hours with knowledge in fixing the
last bugs?

I'm not entirely sure. I just observe that this doesn't seem to be
making much progress.

I think it's one of the drawbacks of taking an ecosystem/libraries
approach instead of a application/framework style approach. An
application or framework typically is an integrated whole that has a
single version number. An ecosystem or set of libraries can be
integrated (which Zope 3 is) but everything evolves at different rates
and there's no single thing to install or talk about.

I'm not saying an ecosystem approach is bad, if that's what Zope 3 wants
to be. I do think that such an approach needs to be supplemented by a
framework approach (and I've been putting work into one way to do that).

If Zope 3 is an ecosystem, a "release" of Zope 3 the ecosystem doesn't
really make much sense. To follow the comparison with Linux
distribution, it's more like a "distribution" of an ecosystem. I'd
therefore suggest that the release of Zope 3.4, if it ever actually
happens, will be the last release of Zope 3 the application server
framework.

I hope that besides Grok, some community will stand up that takes a less
radical approach to building an application server on top of the Zope 3
ecosystem. People having existing applications in Zope 3 to maintain
(like myself with the Document Library) will have a need for it, and
Grok doesn't suit everyone's tastes anyway, especially people
comfortable with existing Zope 3 practices. As I said elsewhere, I
suggest we call this project not "Zope 3" but something else, to avoid
confusion with the Zope ecosystem (and also to avoid implying it's the
clear successor to Zope 2. I think we can safely say by now that's not
how history went).

Regards,

Martijn

Stephan Richter

2007-10-07 23:02:21 UTC

Post by Martijn Faassen
I'm not saying an ecosystem approach is bad, if that's what Zope 3 wants
to be. I do think that such an approach needs to be supplemented by a
framework approach (and I've been putting work into one way to do that).

Why? I have no need for anything on top of Zope 3. I just need to have stable
sets of packages.

Post by Martijn Faassen
If Zope 3 is an ecosystem, a "release" of Zope 3 the ecosystem doesn't
really make much sense. To follow the comparison with Linux
distribution, it's more like a "distribution" of an ecosystem.

I don't understand that sentence.

Post by Martijn Faassen
I'd
therefore suggest that the release of Zope 3.4, if it ever actually
happens, will be the last release of Zope 3 the application server
framework.

That would be really bad. Who defines stable sets then? Again, I think there
is absolutely no need for another framework on top of Zope 3.

Regards,
Stephan

--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training

Martijn Faassen

2007-10-08 11:57:20 UTC

Why? I have no need for anything on top of Zope 3. I just need to have stable
sets of packages.=

Because we have endless confusion between Zope 3 the ecosystem and Zope
3 the web application framework. If you go and make a separate Zope 3
the web application framework and you can get away with just writing a
buildout.cfg, then be happy. Just call it something else, as otherwise
people will confuse it a lot with the exploded Zope 3 libraries. They
don't *have* to use this particular web framework in order to use Zope
3. They can also use Zope 2, or Grok.

I don't understand that sentence.

If Zope 3 is like a linux distribution, we'll have to manage it like a
linux distribution does. A linux distribution has releases, but it
doesn't have releases of something individual you install and run. It
just has semi-frozen ecosystems being released once every while.

You can't do this and at the same time manage Zope 3 as an application,
I think. It's an either-or.

Post by Martijn Faassen
I'd
therefore suggest that the release of Zope 3.4, if it ever actually
happens, will be the last release of Zope 3 the application server
framework.

That would be really bad. Who defines stable sets then? Again, I think there
is absolutely no need for another framework on top of Zope 3.

The stable sets for the web application server are defined by those
people who develop the Zope 3 web application server. The people who
develop the web application server make choices that other users of Zope
3 technology might not make: perhaps to use Twisted for a web server.
Perhaps to use paste for configuration, or *not* use paste for
configuration. They might at some point decide that z3c.form is the
default form framework that is documented in Zope 3 tutorials, instead
of zc.formlib.

Hopefully the users of the Zope 3 technology can work together on
defining stable sets, but probably not for the entire range: Grok has
some extra dependencies that Zope 2 may not have, and vice versa, for
instance.

The Zope 3 web application server is not primarily what the Zope 3
project appears to be developing. I strongly suspect there are more
users of Zope 3 technology within the Plone community than outside it,
for instance. If the Zope 3 project cared about developing a web server,
you'd think it would do a somewhat better job presenting it to the
outside world on a web page, and that there'd be more people to actually
make releases?

Regards,

Martijn

Stephan Richter

2007-10-08 14:49:47 UTC

Post by Martijn Faassen
Because we have endless confusion between Zope 3 the ecosystem and Zope
3 the web application framework.

For me it is exactly the same. Zope 3 is a Web application server. Zope 2 uses
many components of the Zope 3 Web application server.

Post by Martijn Faassen
If you go and make a separate Zope 3
the web application framework and you can get away with just writing a
buildout.cfg, then be happy. Just call it something else, as otherwise
people will confuse it a lot with the exploded Zope 3 libraries. They
don't *have* to use this particular web framework in order to use Zope
3. They can also use Zope 2, or Grok.

But I consider the crafted set of libraries the "Zope 3 application server".
Whether you use just some of its components, choose a different WSGI server,
or exchange functionality is totally irrelevant; it just proves how good the
Zope 3 application server is.

Post by Martijn Faassen
If Zope 3 is like a linux distribution, we'll have to manage it like a
linux distribution does. A linux distribution has releases, but it
doesn't have releases of something individual you install and run. It
just has semi-frozen ecosystems being released once every while.

Right, and I think this is what we need to do for Zope 3. This is what
download.zope.org/zope3.4 and the KGS idea is all about.

Post by Martijn Faassen
You can't do this and at the same time manage Zope 3 as an application,
I think. It's an either-or.

I have never defended the idea of a Zope 3 application. In fact, a lot of my
efforts in Zope 3 right now are to get rid of the ZMI in my projects.

application server != application

Post by Martijn Faassen
The stable sets for the web application server are defined by those
people who develop the Zope 3 web application server. The people who
develop the web application server make choices that other users of Zope
3 technology might not make: perhaps to use Twisted for a web server.
Perhaps to use paste for configuration, or *not* use paste for
configuration. They might at some point decide that z3c.form is the
default form framework that is documented in Zope 3 tutorials, instead
of zc.formlib.

Thus, the KGS should include all packages the sub-communities care about. For
example download.zope.org/zope3.4 has both z3c.form and zope.formlib. Again,
we should look at Linux distributions. They have a very healthy process for
including and excluding packages. The community makes requests of packages to
be included, then they are tested and if all is well, they are added to the
next version of the distribution. If a package looses support and becomes so
outdated that it does not work with the latest set of packages anymore, it is
removed.

Post by Martijn Faassen
Hopefully the users of the Zope 3 technology can work together on
defining stable sets, but probably not for the entire range: Grok has
some extra dependencies that Zope 2 may not have, and vice versa, for
instance.

Well, I would like to have a KGS that can be used by all user projects,
including Zope 2 and Grok. If any of the user projects want to nail
additional versions, they can do so and I would be glad helping to develop
technology to make this happen.

Post by Martijn Faassen
The Zope 3 web application server is not primarily what the Zope 3
project appears to be developing. I strongly suspect there are more
users of Zope 3 technology within the Plone community than outside it,
for instance. If the Zope 3 project cared about developing a web server,
you'd think it would do a somewhat better job presenting it to the
outside world on a web page, and that there'd be more people to actually
make releases?

I think we have very different definitions of application server. For me web
server != application server. In Zope 3's case I could care less what the Web
server is. For me, the publisher is really the server component.

Regards,
Stephan

--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training

Martijn Faassen

2007-10-08 16:07:02 UTC

Hey,

Stephan, I tried to reply to your points but I realized I was getting
lost in a sea of semantics and that it wasn't useful.

I meant to write "web application server", but I didn't write the word
"application", sorry.

I'd like to see a separation between what you consider to be not only
closely allied but *identical*: the pool of Zope 3 components, managed
together by *all* the communities that make use of Zope 3
technologies, and the Zope 3 application server, where you go to some
web site explaining what it's all about, and then download and install
and get some script for to start some webserver so you can access it,
read tutorials on and try to see "Hello world" on your screen with.

Of course since to you there is no difference between the two, it gets
hard to communicate.

The communities that use Zope 3 technologies all have an interest in
improving the common components, and also in distributing such common
components for reuse. Our collective community could be called the
Zope community, where Zope is like a Linux distribution that different
systems make use of, with the major difference that we actually
develop most of those packages within the community instead of mostly
reusing what's developed outside.

The Zope community has always been in the business of building
software that can be used to build (web) applications. For years now
we haven't had a single unified piece as we did when Zope 2 was
(almost) the only game in town in the Zope world.

These days, we have a whole host of related pieces that people can
download and install and build software on top of. We have Zope 2,
Zope 2 + Five, Zope 2 + CMF, Zope 2 + CMF + Five, original Zope 3, and
Zope Grok. What's more we've had things build on top of this that also
have aspects of frameworks or platforms, such as Plone or Silva.

We've also always had components and systems that we use in our
application servers, but could also be used separately: bobo, the
ZODB, zope.interface, buildout, zope.pagetemplate.

We don't have a linear evolution of a single platform at all, even
though for a while we first intended and then pretended it was so with
the naming of Zope 2 and Zope 3. We've stopped pretending for a while,
I think. Instead, we have different communities that share underlying
philosophies and technologies, and interact with each other within the
Zope community. The one thing all these systems have had in common for
the last years is that they all share Zope 3 technologies. If anything
is the Zope platform it's this pool of Zope 3 technologies.

The Zope development community is about the Zope platform. How to
improve the platform? We've done this by improving them, adding to
them, making them easier to evolve independently, supplementing them
with technology like eggs and buildout, testing them, and making them
easier to manage.

The Zope community is also about the things we build on top of the
Zope platform: Zope 2, Zope 3, CMF, Grok, etc. The nice thing about
these is that they make choices for you. If you use Zope 2.10 today,
you know what you have to deal with and you can rely on what you deal
with. With Zope 3.3 it was the same story, and so it is for Grok.
These we can market and offer for download and install. These things
is what we can write tutorials for.

So, the Zope community is a community of communities that are tied
together by their common interest in the Zope platform, on top of
which they build applications, web application frameworks and web
application servers. The Zope platform allows you to roll your own
software directly on top of it if you'd like, but typically you'd make
use of one of the pre-packaged varieties such as Zope 2, Zope 3 or
Grok. All of them are Zope.

Regards,

Martijn

Stephan Richter

2007-10-08 23:33:57 UTC

Post by Martijn Faassen
Stephan, I tried to reply to your points but I realized I was getting
lost in a sea of semantics and that it wasn't useful.

I agree

Post by Martijn Faassen
I'd like to see a separation between what you consider to be not only
closely allied but *identical*: the pool of Zope 3 components, managed
together by *all* the communities that make use of Zope 3
technologies, and the Zope 3 application server, where you go to some
web site explaining what it's all about, and then download and install
and get some script for to start some webserver so you can access it,
read tutorials on and try to see "Hello world" on your screen with.
Of course since to you there is no difference between the two, it gets
hard to communicate.

Exactely. Because it might not be so easy to just click and go. Let's take an
example, because I think it makes it more clear.

For me z3c.formdemo is a good example of a small Zope 3 application. It is
built on top of the Zope 3 Web application server. But in order to get it
working, I did not have to install anything special. I just use the
libraries.

*I do not want to add technology just to match desired terminology.*

Regards,
Stephan

--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training

Martijn Faassen

2007-10-10 12:35:40 UTC

Hey,

Stephan Richter wrote:
[snip]

Post by Stephan Richter
For me z3c.formdemo is a good example of a small Zope 3 application. It is
built on top of the Zope 3 Web application server. But in order to get it
working, I did not have to install anything special. I just use the
libraries.

Sure, when I use Grok I just "use the libraries" too. There's a
particular library in my selection called 'grok' that's not there for
you, but you're probably using other libraries that are not there for me
too.

So how does what you do differ from what I do? If it doesn't, I'd be
using Zope 3 in the exact same way you do, which isn't true, right?

Regards,

Martijn

Jim Fulton

2007-10-08 12:03:18 UTC

I'm not sure that "library" or "collection of libraries" is the right
term for what we want to be. I think we've been using it because it
stands in sharp contrast to "application", which, BTW, isn't exactly
what Zope 2 is. I think these terms were useful to make some points,
but neither is accurate. FWIW, I have a fairly open mind on this
topic. Lots of good points are being made. :)

Jim

Post by Martijn Faassen
[snip]

Do you have a sort explanation on what is the missing resource? Is it,
as it was for 3.3, lack of people-hours with knowledge in fixing the
last bugs?

I'm not entirely sure. I just observe that this doesn't seem to
be making much progress.

I think it's one of the drawbacks of taking an ecosystem/libraries
approach instead of a application/framework style approach. An
application or framework typically is an integrated whole that has
a single version number. An ecosystem or set of libraries can be
integrated (which Zope 3 is) but everything evolves at different
rates and there's no single thing to install or talk about.
I'm not saying an ecosystem approach is bad, if that's what Zope 3
wants to be. I do think that such an approach needs to be
supplemented by a framework approach (and I've been putting work
into one way to do that).
If Zope 3 is an ecosystem, a "release" of Zope 3 the ecosystem
doesn't really make much sense. To follow the comparison with Linux
distribution, it's more like a "distribution" of an ecosystem. I'd
therefore suggest that the release of Zope 3.4, if it ever actually
happens, will be the last release of Zope 3 the application server
framework.
I hope that besides Grok, some community will stand up that takes a
less radical approach to building an application server on top of
the Zope 3 ecosystem. People having existing applications in Zope 3
to maintain (like myself with the Document Library) will have a
need for it, and Grok doesn't suit everyone's tastes anyway,
especially people comfortable with existing Zope 3 practices. As I
said elsewhere, I suggest we call this project not "Zope 3" but
something else, to avoid confusion with the Zope ecosystem (and
also to avoid implying it's the clear successor to Zope 2. I think
we can safely say by now that's not how history went).
Regards,
Martijn
_______________________________________________
Zope3-dev mailing list
Unsub: http://mail.zope.org/mailman/options/zope3-dev/jim%40zope.com

--
Jim Fulton
Zope Corporation

Tres Seaver

2007-10-08 20:09:40 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I'm not sure what you mean with that,

I mean we need to decide what a release is.

Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"

Frozen Releases
- ----------------

A frozen release would consist of:

- A single, "frozen" KGS (index pages cannot change after release).

- Scripts / tools which target that KGS, and create from it a
standalone environment whcih includes the selected diestributiongs
for each project in the KGS. E.g.:

o An easy_install'able meta-egg with pinned versions, referencing
the KGS index as 'index_url'. The egg would also include scripts
and other entry points used to configure the platform environment
(e.g., install zope.conf and site.zcml from skeletons, etc.)

o A bootstrappable zc.buildout with "pinned" versions for all
packages, or which relies on the frozen meta-egg for pinning.

o A 'virtualenv' derivative, again probably leveraging the "frozen"
egg.

It would probably be fairly straightforward to generate the "meta" egg
given the manifest. In fact, the "meta" egg should probably load the
pinned versions from a config file which was also used to generate the
frozen index page.

These releases would be useful for:

- Testing third-party packages which extend the platform.

- Reproducing bugs reported against the platform

- Giving to newbies as a "safe" starting point.

The public versions of them would probably *not* be good for use in
production deployments, because they cannot be updated with bugfixes.
Some users might maintain their own versioned frozen releases for such
uses, and handle updates by re-installing and migrating their data).

Such a release would be analogous to a bootable ISO for a Linux
distribution: if you run from the CD, you *know* what software is
present, without any question.

I would note that the script I posted a week or so ago for building
package index pages from a directory full of source distributions would
be a perfectly adequate tool for constructing such a KGS: simple copy
or link all the "frozen" distributions into a directory and run the
script. Once you've verified that the installation regime works from
the generated files, you *never touch them again*.

Updatable Platform Releases
- ---------------------------

An updatable platform release would consist of:

- A KGS whose index pages were manually updated from time to time with
carefully-selected new distributions of existing packges.

- An installation regime, as above, which uses the KGS as its
'index_url', but *pins no packages* (whether in the "meta" egg,
buildout.cfg, or whatever).

o This regime should also contain / bootstrap scripts which couls
be used to do automated updates from the KGS, like 'yum update'
/ 'apt-get upgrade'.

In this case, generating the "meta egg" (or equivalent) should be
unneeded: that egg could just be managed within the KGS itself. In a
typical environment, the meta egg would likely never be updated all all
(because it contains no software beyond the parts used to generate the
environment).

Such a relase would be analogous to an installed Linux distribution,
with update repositories pre-configured.

Maintaining the KGS in this case is harder, and could probably use a
little more tooling. Once we have the tools, then tweaking them to
allow generating a "frozen" release will be simple. In that mode, the
two flavors of release here could be thought of as like "tags" and
"branches" in the CVS sense (not SVN, which doesn't really have tags).

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

Stephan Richter

2007-10-08 23:49:16 UTC

Post by Tres Seaver
Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"
Frozen Releases
----------------

I started commenting this section until I saw the one below. I personally do
not see much benefit from the frozen release. That said, it would be trivial
to create one from the updateable version below. I have already scripts for
this, which are checked in as part of Jim's PyPI mirror tool.

Post by Tres Seaver
Updatable Platform Releases
---------------------------
?- A KGS whose index pages were manually updated from time to time with
? ?carefully-selected new distributions of existing packges.
?- An installation regime, as above, which uses the KGS as its
? ?'index_url', but *pins no packages* (whether in the "meta" egg,
? ?buildout.cfg, or whatever).
? ?o This regime should also contain / bootstrap scripts which couls
? ? ?be used to do automated updates from the KGS, like 'yum update'
? ? ?/ 'apt-get upgrade'.

This is pretty much done. See http://download.zope.org/zope3.4. I have checked
in the tools in zc.mirrorcheeseshopslashsimple. Buildout itself serves as an
equivalent to ``yum update`` or ``apt-get upgrade``. You can simply say
``./bin/buildout``. Without the "-N" option it will fetch the latest version,
but the KGS guarantees that this will at most be a bug fix.

Post by Tres Seaver
In this case, generating the "meta egg" (or equivalent) should be
unneeded: ?that egg could just be managed within the KGS itself. ?In a
typical environment, the meta egg would likely never be updated all all
(because it contains no software beyond the parts used to generate the
environment).

Yep.

Post by Tres Seaver
Such a relase would be analogous to an installed Linux distribution,
with update repositories pre-configured.

Exactely!

Post by Tres Seaver
Maintaining the KGS in this case is harder, and could probably use a
little more tooling. ?Once we have the tools, then tweaking them to
allow generating a "frozen" release will be simple. ?In that mode, the
two flavors of release here could be thought of as like "tags" and
"branches" in the CVS sense (not SVN, which doesn't really have tags).

Yep, I agree. The usefulness of tags other than for generating a full release
is questionable in my opinion, but I still agree with your ananlisys. :-)

Regards,
Stephan

--
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training

Martijn Faassen

2007-10-10 12:45:15 UTC

Post by Tres Seaver
Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"
Frozen Releases
----------------

I don't understand why you believe you couldn't benefit from a frozen
release.

If you don't have a frozen release, you always run the following risk:

* you release your code to someone else (another customer, or another
developer)

* after this, someone uploads a new bugfix release for an egg which
causes your code to break

* the customer, or another developer, installs the code, and it breaks,
where it used to work when you released it.

This risk can never be completely eliminated without frozen releases.

Regards,

Martijn

Brian Sutherland

2007-10-10 15:25:56 UTC

Post by Tres Seaver
Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"
Frozen Releases
----------------

A frozen release is very useful for some people (as long as it also gets
security updates).

--
Brian Sutherland

Tres Seaver

2007-10-10 15:55:38 UTC

Post by Brian Sutherland

Post by Tres Seaver
Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"
Frozen Releases
----------------

A frozen release is very useful for some people (as long as it also gets
security updates).

That would be like being "a little bit pregnant." What you describe is
an "updatable release" with a particular policy for how the updates are
accepted into the KGS: you would allow new distributions for existing
packges only where they included security fixes. Such a KGS would
require meaintenance effort:

- The maintainer would need to monitor devlopment of "upstream"
projects, in order to know that new distributions had been released.

- In some cases, the maintainer might need to create separate
distributions for some upstream eggs, in order to strip out changes
unrelated to a given security fix.

That kind of KGS / release could certainly be useful, but it wouldn't
be, nor serve the same goals as, a frozen release.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

Tres Seaver

2007-10-10 17:10:07 UTC

Post by Brian Sutherland

Post by Tres Seaver
Presuming agreement on the "known good set" (KGS) term, I would think
that we have two candidates for what makes up "platform releases"
Frozen Releases
----------------

A frozen release is very useful for some people (as long as it also gets
security updates).

Roger Ineichen

2007-10-09 03:19:35 UTC

Hi Tres

Cc: zope3-dev Development
Betreff: [Zope3-dev] Re: Zope 3 releases?

[...]

- A single, "frozen" KGS (index pages cannot change after release).

Can we use a flag on the server side e.g. in the index page, so
nobody is able to upload files if we use automated tools
for the upload?

Or can we use a different url which we can't upload after
the freeze has be done. Like we do in subversion at least in some
svn clients there will be a not if you try to commit to a url which
contains the /tag/ part.

[...]

I would note that the script I posted a week or so ago for
building package index pages from a directory full of source
distributions would be a perfectly adequate tool for
constructing such a KGS: simple copy or link all the
"frozen" distributions into a directory and run the script.
Once you've verified that the installation regime works from
the generated files, you *never touch them again*.

Do you think the directory should be a subversion tag
which makes it reproducable?

[...]

Roger Ineichen

Tres.

Martijn Faassen

2007-10-10 12:41:18 UTC

Hey,

Tres Seaver wrote:
[snip]

Post by Tres Seaver
Frozen Releases
- ----------------
- A single, "frozen" KGS (index pages cannot change after release).

[snip]

With Grok we're now using such a versions list with buildout, using the
buildout extends mechanism. This has two advantages:

* you can just use the cheeseshop. (has the drawback we don't cache
releases though, so a cheeseshop release cache (without index page)
might be useful in the future to look up missing versions)

* you can combine this list with others, which allows you to build
frameworks on frameworks. I consider this is an essential use case.

In the index page approach you'd have to *copy* the KGS frozen list for
a later Grok release, while with the Grok approach I believe we can just
point at a URL. If I then have framework X which builds on Grok I'd need
to *copy* this list again. If I then want to upgrade to a newer version
of Grok, I'd need to do the whole thing again, which means that besides
one single big list I'd also need to maintain the smaller list for Grok
and my framework X. The buildout extends mechanism seems to offer a way
out that's there today.

Please don't ignore this working approach for frozen releases?

Regards,

Martijn

Tres Seaver

2007-10-08 20:13:20 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I'm not sure what you mean with that,

I mean we need to decide what a release is.

Ignas Mikalajunas

2007-10-12 11:54:29 UTC

Post by Jim Fulton
- We need to decide what a Zope 3 release is (or maybe multiple
flavors). I favor copying the linux experiences, but have an open mind.

I'm not sure what you mean with that,

I mean we need to decide what a release is.

Having a list of "promises" with the decision of what a release is
would be nice too. Tarball release was (at least i thought that it
was) promising that:

* You will be able to migrate from 3.3 to 3.4 without any errors, only
with a lot of deprecation warnings.
* 3.3 will only have bugfixes added, not new features.
* 3.3 will work.
* Your database will evolve from 3.3 to 3.4.

I am a bit wary that skipping a tarball release and going directly to
eggs might cause evolution and backwards compatibility problems for
anyone that was using the 3.3 tarball. I only managed to migrate from
3.3 to 3.4 eggs through 3.4.0b1 tarball, because SchoolTool just
didn't work with eggs without a significant set of changes to the
code.

So releasing a tarball and then working towards an egg based 3.5 that
will have a migration path from 3.4 tarball might take less time, than
creating a set of 3.4 eggs that work with 3.3 applications.

That is my opinion too.

Well - if it will be possible to go from Zope 3.3 that got released in
Ubuntu to a set of zope libraries packaged and installed separately,
it's good enough for me. Though having no intermediate step might add
work for the people who will do packaging of zope for various linux
distributions. And yes, I understand that while building a web service
one does not need Zope 3 available in Linux distributions, but as I
really want to build an application and most Linux users install
applications using package managers provided by their distributions, I
would really like seeing Zope3 released in a manner that can be
packaged.

Ignas

Christian Theune

2007-10-11 15:42:25 UTC