[Zope3-dev] Known-good-sets problem

[Jim Fulton]

I discussed this a bit this afternoon with Stephan and we came up
with an idea that we think might help. Stephan is going to try to
prototype it. I'll try to explain it.

The basic idea is to provide a custom index. There will be such an
index for each "known good set" (KGS). An example of a KGS would be
the KGS corresponding to the Zope 3.4 release. A KGS would have a
set of controlled projects. A KGS index will have manually-managed
project pages for all controlled projects. For other projects, it
will mirror PyPI.

The prototype will build on my cheeseshop mirroring software. We
will add a controlled projects list as configuration data. The
mirroring software will ignore updates for controlled projects. This
is a very small change to existing simple software. (The controlled
project directories could be managed by either editing index pages or
by placing approved distros into a server directory.) The custom
index will be a static web site.

With this in place, we can establish KGSs and, for controlled
projects, the index pages will only be updated when a distribution
for a known project has been carefully vetted.

Users can set up buildout or easy_install to use the specific KGS as
their index server. Each KGS will have a release manager who will be
responsible for maintaining the KGS.

We will also create a buildout that tests packages in the KGS. When
one wants to test a change to a core package, they would:

- check out the buildout
- maybe change the index option to point to a particular KGS
- check out the project(s) they want to test and configure them as
develop eggs
- run the buildout test script.

Hopefully, this will give us much greater stability than we've had up
to now.

--
Jim Fulton
Zope Corporation

[Stephan Richter]

I discussed this a bit this afternoon with Stephan and we came up ?
with an idea that we think might help. ?Stephan is going to try to ?
prototype it. ?I'll try to explain it

Hi everyone,

the first version of the tools required to develop KGS's is now done. I simply
added them to the ``zc.mirrorcheeseshopslashsimple``. That said, here is what
we have:

1. There is a new index located at http://download.zope.org/zope3.4.

2. This index contains all PyPI packages.

3. There is a file called "controlled-packages.cfg" that has a list of
projects for which we define a set of versions that are known stable within
this index. The projects' pages are rewritten to only link to the stable
versions.

4. There is also a "buildout.cfg" that is generated from the controlled
packages config file. You can download it and use it to test the complete set
of packages. (There are currently test failures!)

The good part about it is that the index works right now giving application
developers some stability. However, there are still some issues:

1. Not all zope.* and zope.app* packages are yet listed. An example
is "zope.viewlet".

2. How many packages should be controlled in this index? I think we should
definitely add packages from z3c and the zc namespace.

3. The workflow of maintaining the KGS is somewhat unclear. Here is a workflow
that works right now:

(a) Install zc.mirrorcheeseshopslashsimple tools.

(b) Download "controlled-packages.cfg" from the set you want to modify or
test.

(c) Use the "generate-buildout" script to create a "buildout.cfg" file.

(d) Without using the KGS index (so you can pickup new versions), run
buildout, so that the "test" script is created.

(e-I) Change "controlled-packages.cfg" to include a new version of a package
and go back to (c).

(e-II) Alternatively, if you want to create a new release of a package to fix
issues in the KGS, you can list the package in "buildout.cfg" as a develop
egg and make a new version entry in "controlled-packages.cfg" when you have
released a new version.

(f) Once you are done, upload "controlled-packages.cfg" again. Note that this
file is not under version control, so be careful to not overwrite someones
work. This is probably a good reason to only have one release manager per
KGS.

Overall I am very pleased with that approach since it addresses our current
instability problems. It also resembles QA methods used by the Linux
distribution creators.

BTW, I shamelessly copied the initial list of stable versions from grok.

Regards,
Stephan

[Benji York]

Post by Stephan Richter
2. How many packages should be controlled in this index? I think we should
definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages? I suppose it
is to let people use those packages with (in this case) Zope 3.4. What
if someone wants Zope 3.4 and Twisted version X and Plone version Y
(just making those up). Perhaps we need a way to refer to several KGS
when constructing an application. Or is one KGS supposed to define a
"universe" of packages known to work together. If so, I would think
there would be no place for non-controlled packages.

[Stephan Richter]

Post by Benji York

Post by Stephan Richter
2. How many packages should be controlled in this index? I think we
should definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages?

Jim mentioned that setuptools only supports one index. :-( If that's not true,
then I would be glad to try a more constrained approach.

Post by Benji York
I suppose it
is to let people use those packages with (in this case) Zope 3.4. ?What
if someone wants Zope 3.4 and Twisted version X and Plone version Y
(just making those up). ?Perhaps we need a way to refer to several KGS
when constructing an application.

If this would be possible, then great!

Regards,
Stephan

[Martijn Faassen]

Post by Stephan Richter

Post by Benji York

Post by Stephan Richter
2. How many packages should be controlled in this index? I think we
should definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages?

Jim mentioned that setuptools only supports one index. :-( If that's not true,
then I would be glad to try a more constrained approach.

Note that the buildout extends mechanism does support extending multiple
other cfg files. I think it should then be possible to reuse multiple
lists. It also doesn't require new server software; we just publish
these lists over HTTP.

Regards,

Martijn

[Martijn Faassen]

Post by Benji York

Post by Stephan Richter
2. How many packages should be controlled in this index? I think we
should definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages? I suppose it
is to let people use those packages with (in this case) Zope 3.4. What
if someone wants Zope 3.4 and Twisted version X and Plone version Y
(just making those up). Perhaps we need a way to refer to several KGS
when constructing an application. Or is one KGS supposed to define a
"universe" of packages known to work together. If so, I would think
there would be no place for non-controlled packages.

A feature to combine lists of versions is definitely needed. I may be a
developer of an application that uses both Grok and KSS, for instance.
This can be solved by me maintaining my own list, but I'd prefer to rely
on two lists that are already there.

That's the application developer's case. The framework developer's case
requires me to publish such combined lists. If I have my own framework
on top of Grok and KSS, I'd like to publish a list that's a combination
of those without having to copy them.

Regards,

Martijn

[Jim Fulton]

Post by Martijn Faassen

Post by Benji York

Post by Stephan Richter
2. How many packages should be controlled in this index? I think
we should definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages? I
suppose it is to let people use those packages with (in this case)
Zope 3.4. What if someone wants Zope 3.4 and Twisted version X
and Plone version Y (just making those up). Perhaps we need a way
to refer to several KGS when constructing an application. Or is
one KGS supposed to define a "universe" of packages known to work
together. If so, I would think there would be no place for non-
controlled packages.

A feature to combine lists of versions is definitely needed. I may
be a developer of an application that uses both Grok and KSS, for
instance. This can be solved by me maintaining my own list, but I'd
prefer to rely on two lists that are already there.
That's the application developer's case. The framework developer's
case requires me to publish such combined lists. If I have my own
framework on top of Grok and KSS, I'd like to publish a list that's
a combination of those without having to copy them.

As I mentioned in my response to Benji, it's tricky to get the
semantics right. IMO, you want a particular KGS to be
authoritative. There isn't anything in setuptools to support this
semantic. You can use the mirroring approach (maybe with some minor
tweaking) to combine KGSs on the server.

Jim

--
Jim Fulton
Zope Corporation

[Jim Fulton]

Post by Benji York

Post by Stephan Richter
2. How many packages should be controlled in this index? I think
we should definitely add packages from z3c and the zc namespace.

What is the motivation to include non-controlled packages? I
suppose it is to let people use those packages with (in this case)
Zope 3.4.

Yes

Post by Benji York
What if someone wants Zope 3.4 and Twisted version X and Plone
version Y (just making those up).

That's why the KGS index is a PyPI mirror of all uncontrolled packages.

Post by Benji York
Perhaps we need a way to refer to several KGS when constructing
an application. Or is one KGS supposed to define a "universe" of
packages known to work together. If so, I would think there would
be no place for non-controlled packages.

The semantics I think we want are kind of tricky. A KGS index needs
to be authoritative for the projects it controls. If we looked in
multiple KGSs, there would need to be semantics for deciding which
was authoritative. setuptools lets you define a single index and
multiple find-link servers. The highest version found on any server
is authoritative. I think supporting multiple KGSs with the right
semantics would be useful, but there isn't a way to do it in
setuptools. We can achieve the same effect on the server. For
example, with this software, you could chain several KGSs together.

Jim

--
Jim Fulton
Zope Corporation

[Martijn Faassen]

Hey,

I'm glad some steps are taken!

What I really don't like about this proposal is that it talks about
updating index pages. If I understand this right, an updated index page
will force everybody that uses this index page into an update. I don't
think this is acceptable. Instead I'd suggest you institute a rule that
index pages should never be changed after initial publication, just like
eggs shouldn't be changed after publication either. Instead, a new index
page should be published for each update. People can then choose to
switch whenever they like.

My main suggestion is to not forget that this needs to be clearly
documented and the document describing which steps to take should be
published.

My main worry is that I don't like maintaining dependency lists on
remote servers somewhere. My intuition is that we should maintain these
lists close to the actual software. With grok we're managing this list
now as a versions.cfg in subversion. Anyway, this is just a worry, and
we'll have to see how this pans out.

We've gone another direction with the Grok project this week, where we
publish a versions.cfg on the web that gets included in a buildout.cfg.
This approach was based on suggestions by yourself to me, and had been
discussed by Philipp on the mailing list earlier, but apparently the
choice was made to go for index pages instead.

So, it's a bit of a shame this extensive work will now be made mostly
redundant by these new efforts. Anyway, we'll be using this for a while
at least and we'll wait and see how this new works pans out.

Regards,

Martijn

[Jim Fulton]

Post by Martijn Faassen
Hey,
I'm glad some steps are taken!
What I really don't like about this proposal is that it talks about
updating index pages. If I understand this right, an updated index
page will force everybody that uses this index page into an update.

Only people who ask for updates.

Post by Martijn Faassen
I don't think this is acceptable.

I think this depends on the use cases. I think most people would
like to get bug fixes. The intent of such a 3.4 index would be to
give people the latest 3.4 updates which should give no new
features. This is intended to mimick what happens with linux package
respositories.

Post by Martijn Faassen
Instead I'd suggest you institute a rule that index pages should
never be changed after initial publication, just like eggs
shouldn't be changed after publication either. Instead, a new index
page should be published for each update. People can then choose to
switch whenever they like.

You could manage an index like that. I don't think that would be
very useful. IMO, if you want to really nail version numbers, then
it would be better to just use meta eggs or buildout version
specifications.

The goal of the index is to give people a stable source of
distributions that are known to work together.

Post by Martijn Faassen
My main suggestion is to not forget that this needs to be clearly
documented and the document describing which steps to take should
be published.

Absolutely. The index is just a mechanism that will enable processes
that actually address the problems we're struggling with,

Post by Martijn Faassen
My main worry is that I don't like maintaining dependency lists on
remote servers somewhere. My intuition is that we should maintain
these lists close to the actual software. With grok we're managing
this list now as a versions.cfg in subversion. Anyway, this is just
a worry, and we'll have to see how this pans out.

That's a valid approach, at least for some problems.

If the software is an application, the best approach IMO is for each
release to specify precisely the versions it's using. If software is
a platform, then more flexibility may be needed, This index idea
seeks to learn from experience with linux distributions.

Post by Martijn Faassen
We've gone another direction with the Grok project this week, where
we publish a versions.cfg on the web that gets included in a
buildout.cfg. This approach was based on suggestions by yourself to
me, and had been discussed by Philipp on the mailing list earlier,
but apparently the choice was made to go for index pages instead.

Yup. Of course, it only works with buildout. At the time, I
remember getting push back from someone that they wanted a solution
that didn't require buildout. A custom index is more open both wrt
packaging technology and to use by different applications. I think
Zope 3 would benefit from a stable repository of eggs that can be
used even if people aren't using Grok or buildout.

Of course, the index technology isn't worth much without solid
processes behind it.

Post by Martijn Faassen
So, it's a bit of a shame this extensive work will now be made
mostly redundant by these new efforts.

I don't think so. Certainly, the detective work you've done figuring
out what's compatibly should be used by whoever assembles a 3.4 index.

Post by Martijn Faassen
Anyway, we'll be using this for a while at least and we'll wait and
see how this new works pans out.

I think the index complements techniques like meta eggs or
buildout.cfgs with nailed versions.

Also, keep in mind that this effort results from a desire to help and
because people care. Be careful about accusing people of not caring
and then dismissing their efforts to help.

I really appreciate the efforts of Philipp and Baiju to help us make
incremental progress on the release process for individual projects.
I think we need to start working (thinking & talking) about the
release process for the larger Zope 3 ecosystem.

Jim

--
Jim Fulton
Zope Corporation

[Martijn Faassen]

[snip]

Post by Jim Fulton

Post by Martijn Faassen
What I really don't like about this proposal is that it talks about
updating index pages. If I understand this right, an updated index
page will force everybody that uses this index page into an update.

Only people who ask for updates.

Yes, but it's important in releases. If I release my application to
someone else, they will be asking for an update while I might never had.
I cannot test this as the updated versions might've been released
*after* I make my application release.

You can argue I should maintain my own index for releases (or do some
other packaging). This would then only be a tool to assist developers.
Then again, developers also share code and could run into similar issues.

Post by Jim Fulton

Post by Martijn Faassen
I don't think this is acceptable.

I think this depends on the use cases. I think most people would like
to get bug fixes. The intent of such a 3.4 index would be to give
people the latest 3.4 updates which should give no new features. This
is intended to mimick what happens with linux package respositories.

Yes, bugfixes is less of a problem than feature releases, if this is
well managed. Since packages evolve at different rates, what is done for
feature releases of packages? Do they end up in the same index or
separate indexes? How to determine when to make a new index?

Let me make the case for bug-for-bug compatibility:

The problems during release as I sketched out above still exist. Granted
for bugfix releases the chances are lower that something breaks than it
is now, but we all know people sometimes disagree about what actually
constitutes a bugfix (or larger change), and people also sometimes have
workarounds or code assumptions that depend on bugs. Bugfixes can break
working code.

Post by Jim Fulton

Post by Martijn Faassen
Instead I'd suggest you institute a rule that index pages should never
be changed after initial publication, just like eggs shouldn't be
changed after publication either. Instead, a new index page should be
published for each update. People can then choose to switch whenever
they like.

You could manage an index like that. I don't think that would be very
useful. IMO, if you want to really nail version numbers, then it would
be better to just use meta eggs or buildout version specifications.
The goal of the index is to give people a stable source of distributions
that are known to work together.

Okay, understood. I can see some utility for this, if only to help me
generate the lists of pinned versions. :)

[snip]

Post by Jim Fulton
If the software is an application, the best approach IMO is for each
release to specify precisely the versions it's using. If software is a
platform, then more flexibility may be needed, This index idea seeks to
learn from experience with linux distributions.

I see some differences between this and a Linux distribution.

A linux distribution releases bugfix releases when:

* there's a security problem

* in some cases when there's something obviously broken in an application

I don't know whether linux distributions commonly release bugfix
releases in *libraries*. Do Linux distributions commonly release bugfix
versions of glibc, or Python libraries? I guess it depends on how
severely the bug impacts important applications.

In our case, we generally release bugfixes to developers of
applications, not users of applications. For security bugfixes, I can
see the argument for getting them out as quickly as possible, even at
the risk of sometimes breaking applications.

What other kinds of bugs do we have? How do they affect developers?

Post by Jim Fulton

Post by Martijn Faassen
We've gone another direction with the Grok project this week, where we
publish a versions.cfg on the web that gets included in a
buildout.cfg. This approach was based on suggestions by yourself to
me, and had been discussed by Philipp on the mailing list earlier, but
apparently the choice was made to go for index pages instead.

Yup. Of course, it only works with buildout. At the time, I remember
getting push back from someone that they wanted a solution that didn't
require buildout. A custom index is more open both wrt packaging
technology and to use by different applications. I think Zope 3 would
benefit from a stable repository of eggs that can be used even if people
aren't using Grok or buildout.

Sure. I don't think having such an index is a bad idea. I'm watching
this experiment with much interest and I think it helps us asking good,
concrete questions about potential problems to refine our story here.

Post by Jim Fulton
Of course, the index technology isn't worth much without solid processes
behind it.

Post by Martijn Faassen
Anyway, we'll be using this for a while at least and we'll wait and
see how this new works pans out.

I think the index complements techniques like meta eggs or
buildout.cfgs with nailed versions.

That's a good point. I can see how much of the process could be shared
between the maintainers of an evolving index and the maintainers of
version lists.

[snip]

Post by Jim Fulton
Also, keep in mind that this effort results from a desire to help and
because people care. Be careful about accusing people of not caring and
then dismissing their efforts to help.

I wasn't intending to dismiss this project at all, I am trying to give
my feedback to it and trying to understand the goals. I understand it a
bit better now. My apologies for saying people don't care.

Post by Jim Fulton
I really appreciate the efforts of Philipp and Baiju to help us make
incremental progress on the release process for individual projects. I
think we need to start working (thinking & talking) about the release
process for the larger Zope 3 ecosystem.

I see this as part of the conversation.

To me personally this issue evolved from "serious" to "problematic" to
"critical" over the last month or so, as I noticed our Grok release
broke over and over again (which sucks if you try to reach out to
beginners), and I got several customers contacting me with buildout
problems. Several other developers also reported such problems. So I got
pretty frustrated over this. I should've shown it less.

Regards,

Martijn

[Jim Fulton]

Post by Martijn Faassen
[snip]

Post by Jim Fulton

Post by Martijn Faassen
What I really don't like about this proposal is that it talks
about updating index pages. If I understand this right, an
updated index page will force everybody that uses this index page
into an update.

Only people who ask for updates.

Yes, but it's important in releases. If I release my application to
someone else, they will be asking for an update while I might never
had. I cannot test this as the updated versions might've been
released *after* I make my application release.

Of course, each update to an index is, in essense a new release of
the index.

...

Post by Martijn Faassen

Post by Jim Fulton

Post by Martijn Faassen
I don't think this is acceptable.

I think this depends on the use cases. I think most people would
like to get bug fixes. The intent of such a 3.4 index would be to
give people the latest 3.4 updates which should give no new
features. This is intended to mimick what happens with linux
package respositories.

Yes, bugfixes is less of a problem than feature releases, if this
is well managed. Since packages evolve at different rates, what is
done for feature releases of packages? Do they end up in the same
index or separate indexes? How to determine when to make a new index?

Those are all good questions that need to be answered. Of the top of
my head, :)

- I imagine there would be an index for each feature release that
gets bug -fix updates.

- There would be an index for the next feature release that includes
stable packages with new features. There would be some requirements
for packages to be deamed ready for the next release.
I assume by this you mean setting fixed versions.

Post by Martijn Faassen
The problems during release as I sketched out above still exist.
Granted for bugfix releases the chances are lower that something
breaks than it is now, but we all know people sometimes disagree
about what actually constitutes a bugfix (or larger change), and
people also sometimes have workarounds or code assumptions that
depend on bugs. Bugfixes can break working code.

I'm not suggesting that setting fixed versions is a bad idea. I
think some projects may choose to go this way. This is a valid
policy decision. Other developers would, IMO, benefit from having a
relatively stable index as a source of new packages. I don't think
this is an either-or decision.

Jim

--
Jim Fulton
Zope Corporation

[Martijn Faassen]

Jim Fulton wrote:
[snip]

Post by Jim Fulton
I assume by this you mean setting fixed versions.

Yes.

[snip]

Post by Jim Fulton
I'm not suggesting that setting fixed versions is a bad idea. I think
some projects may choose to go this way. This is a valid policy
decision. Other developers would, IMO, benefit from having a relatively
stable index as a source of new packages. I don't think this is an
either-or decision.

I agree it isn't. I just wanted to make the case be heard in this
discussion in case people hadn't seen the arguments yet.

Regards,

Martijn

[Lennart Regebro]

Post by Jim Fulton
Only people who ask for updates.

Which by default is every time somebody runs a buildout, right?

Post by Jim Fulton
I think this depends on the use cases. I think most people would
like to get bug fixes. The intent of such a 3.4 index would be to
give people the latest 3.4 updates which should give no new
features. This is intended to mimick what happens with linux package
respositories.

Ah, good point. But I think what Martijn is saying is that when we
come with new *features* this should be made into a new page. Which
would then be Zope 3.5, or something. Or did I misunderstand him?

But you are right, small bugfixes, and in particular security fixes,
should indeed go into the already existing page. Which means all you
would need to get a security fix would be to run buildout, possibly
forcing it to check for new versions if that is set to false by
default. That would indeed be a very nice feature.

[Jim Fulton]

Post by Lennart Regebro

Post by Jim Fulton
Only people who ask for updates.

Which by default is every time somebody runs a buildout, right?

Yes

Post by Lennart Regebro

Post by Jim Fulton
I think this depends on the use cases. I think most people would
like to get bug fixes. The intent of such a 3.4 index would be to
give people the latest 3.4 updates which should give no new
features. This is intended to mimick what happens with linux package
respositories.

Ah, good point. But I think what Martijn is saying is that when we
come with new *features* this should be made into a new page. Which
would then be Zope 3.5, or something. Or did I misunderstand him?

I don't know if you did or not. I'll just note that different indexes
might have different update policies. A Zope 3.4 index should only
get bug fixes.

Post by Lennart Regebro
But you are right, small bugfixes, and in particular security fixes,
should indeed go into the already existing page. Which means all you
would need to get a security fix would be to run buildout, possibly
forcing it to check for new versions if that is set to false by
default. That would indeed be a very nice feature.

I think so.

Jim

--
Jim Fulton
Zope Corporation

[Martijn Faassen]

Hey,

Another point of feedback:

I saw Stephan's mail on a (partially new) toolchain and somewhat
extensive workflow on using it. I'm a bit surprised a toolchain is
necessary and that the workflow is so involved. With Grok's approach
using extends in buildout, we can just publish such a list on a dumb
website without any tools whatsoever.

Since you're effectively mirroring the cheeseshop anyway, why not rely
on the cheeseshop entirely and just publish the version lists? Could
someone point out the benefits of this approach over just publishing
versions? The one I can think of is that you can potentially retain
packages that were for some reason removed from the cheeseshop, but even
that is theoretical right now, right? As far as I understand, the
buildout versions list approach would allow one to combine lists
together as well, a feature lacking in the current approach.

I'm glad to see the effort to run tests over the lists of packages. This
was something missing from our Grok story and one that I was worried about.

Regards,

Martijn

[Tres Seaver]

Post by Martijn Faassen
Hey,
I saw Stephan's mail on a (partially new) toolchain and somewhat
extensive workflow on using it. I'm a bit surprised a toolchain is
necessary and that the workflow is so involved. With Grok's approach
using extends in buildout, we can just publish such a list on a dumb
website without any tools whatsoever.

I'll agree that I don't see a need for much of a toolchain.
Effectively, the index pages are just a set of N+1 static HTML pages
(where N is the number of distutils "projects" managed by the index)
with mirrored copies of all the distributions in some well-known (and
guarded ;) location.

Post by Martijn Faassen
Since you're effectively mirroring the cheeseshop anyway, why not rely
on the cheeseshop entirely and just publish the version lists?

If you don't mirror the actual distributions, then you are at the mercy
of the project owner on the cheeseshop, who is free to remove or (worse)
update-in-place an egg you rely on.

This is why Debian mirrors the "pristine" tarballs of packges, and why
RPM bundles them inside the SRPMs: *other* people and their software
distribution mechanisms can't be relied on when repeatability is your goal.


- -Tres.-
===================================================================
Tres Seaver [phone] ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[Tres Seaver]

Post by Martijn Faassen
Hey,
I saw Stephan's mail on a (partially new) toolchain and somewhat
extensive workflow on using it. I'm a bit surprised a toolchain is
necessary and that the workflow is so involved. With Grok's approach
using extends in buildout, we can just publish such a list on a dumb
website without any tools whatsoever.

I'll agree that I don't see a need for much of a toolchain.
Effectively, the index pages are just a set of N+1 static HTML pages
(where N is the number of distutils "projects" managed by the index)
with mirrored copies of all the distributions in some well-known (and
guarded ;) location.

Post by Martijn Faassen
Since you're effectively mirroring the cheeseshop anyway, why not rely
on the cheeseshop entirely and just publish the version lists?

If you don't mirror the actual distributions, then you are at the mercy
of the project owner on the cheeseshop, who is free to remove or (worse)
update-in-place an egg you rely on.

This is why Debian mirrors the "pristine" tarballs of packges, and why
RPM bundles them inside the SRPMs: *other* people and their software
distribution mechanisms can't be relied on when repeatability is your goal.


- -Tres.-
===================================================================
Tres Seaver [phone] ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[Jan-Wijbrand Kolman]

I discussed this a bit this afternoon with Stephan and we came up with
an idea that we think might help. Stephan is going to try to prototype
it. I'll try to explain it.

snip

Hopefully, this will give us much greater stability than we've had up to
now.

For what it is worth, what follows is the report I send yesterday to the
Grok mailinglist that is the result of trying to "pin down" a known
working set to rely on for Grok.

I personally like the simplicity of the described solutions and the fact
the we really did not need to *invent* anything new, but were able to
use the tools (buildout specifically) that are there already.

I hope this is of any use, and feedback from this list is appreciated.

kind regards,
jw


-------- Original Message --------
Subject: Pinning Down Versions
Date: Fri, 05 Oct 2007 19:14:16 +0200
From: Jan-Wijbrand Kolman <***@gmail.com>
To: grok-***@zope.org
Newsgroups: gmane.comp.web.zope.grok.devel

===============
Pinned versions
===============

During the Neanderthal Grok Sprint, Luciano and I have been working
on "pinning down versions" to protect Grok development and deployment
from broken, incompatible and conflicting egg releases of all the
dependencies.

The first thing we did is to come up with and test (on Linux, OS X
and Windows) a list of known working egg versions. You can see the
current list here:

http://svn.zope.org/*checkout*/grok/trunk/versions.cfg

With this list of versions, we updated Grok's `buildout.cfg` to:

1) use the `versions.cfg` by way of the `extends` directive.

2) point the `versions` directive to the versions section that is
defined in the `versions.cfg` file.

What this now means for the development *of* Grok and the development
*with* Grok is described below.

**NOTE:** Now that we have a set of versions that actually work, we
can evolve this to something with as much final releases in at as
possible.

For developers *of* Grok
=========================

The `buildout.cfg` of Grok now use the `versions` and `extends`
directive like so::

[buildout]
....
extends = versions.cfg
versions = versions # being defined in the versions.cfg file
....

Now, when running `./bin/buildout` it will make sure exactly those
versions of the egg dependencies are downloaded and installed for
this Grok buildout.

During Grok's lifecycle newer versions of the egg dependencies might
be released. If you want to test such a newer version, you could
temporarily modify the `buildout.cfg` and override the `versions`
section like so::

[buildout]
extends = versions.cfg
versions = versions # being defined in the versions.cfg file
....

[versions]
# whatever specific egg version is defined in this section,
# will override the version in the versions.cfg
zope.i18nmessageid = 3.5.0

One other way to do this is to create a temmporary config file,
say `dev.cfg` and experiment with newer eggs versions like so::

[buildout]
extends = buildout.cfg

[versions]
# whatever specific egg version is defined in this section,
# will override the version in the versions.cfg
zope.i18nmessageid = 3.5.0

Make sure you use this config file when running the buildout command
like so::

$ ./bin/buildout -c dev.cfg

If the newer egg version proves to be compatible, the newer version
number can be put in the `versions.cfg` and checked in. We think it
is very important to make sure the newer egg version *and its
dependencies* are to be available on at least Windows too.

Whenever a Grok release tag is made, we obviously have a tagged fixed
`versions.cfg` file as well. Besides the regular steps needed to make
an actual Grok release, extra steps will be required to support
grokproject (the reason is explained in the following section):

1) the tagged `versions.cfg` needs to be uploaded to
http://grok.zope.org/releaseinfo/grok-[versionnumber].cfg

2) the version of the Grok egg needs to be added to the
`grok-[versionnumber].cfg` file, e.g. `grok = 0.11`.

3) the contents of http://grok.zope.org/releaseinfo/current
needs to be updated to contain the name of the current
'released' Grok versions info file. E.g. `grok-0.11.cfg`.

For developers *with* Grok
==========================

We think it is reasonable to assume that users of Grok, in other
words application developers that use Grok as a basis, are using
grokproject for setting up the initial project structure they'll use.
At least they should be strongly encouraged to do so.

We should also support upgrading an already existing project that has
been created with grokproject, i.e. when a new Grok has been released.

Initial project creation
------------------------

For the grokproject user, nothing will change really. He will issue
the grokproject command and this will setup an project structure for
him.

What the new grokproject will do by default is this:

1) fetch the name of the current versions file from
http://grok.zope.org/releaseinfo/current

2) add to the `buildout.cfg` an `extends` directive pointing to
the URL of the named versions file. E.g.
http://grok.zope.org/releaseinfo/grok-0.11.cfg

3) add the `versions` directive pointing to the `[versions]`
section in the named versions file

Now when the buildout command is run, exactly those versions of the
eggs will get installed.

When running grokproject this behaviour can be overridden. The
application developer can choose to specify the URL pointing to
custom versions file like so::

$ grokproject \
--version-info-url=http://hostname/some/path/versionsfile.cfg

The `extends` directive will then point to that URL.

**NOTE:** the implementation for the "new" grokproject lives on the
"jw-luciano-use-versions-file" branch until it is merged, probably
somewhere next week.

Overriding versions of particular eggs
--------------------------------------

It is likely that the application developer introduces more
dependencies on other software. He should of course add this
dependecy in his `setup.py file`. If he wants to make sure that a
specific version of the dependency is used when deploying the
project, he can add this specification to his `buildout.cfg` like so::

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
versions = versions
....

[versions]
spam = 1.2.3

Likewise, if the application developers wants to override the version
of one of the eggs grok depends on he can adjust his `buildout.cfg`
to look like this::

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
versions = versions
....

[versions]
zope.i18nmessageid = 3.5.0 # will override the original spec

By the way, users of grokproject should be encouraged to let the Grok
developers know about wanting to use a newer version of a current
Grok dependency. With this information, the Grok developers can
consider this newer version to be specified - "blessed" - for the
upcomming release of Grok.

Upgrading to a new Grok release
-------------------------------

Whenever an application developer is ready to upgrade his project to
a new version of Grok, he only needs to change the URL that points to
the versions file to the newest versions file::

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.99.cfg
versions = versions
....

Then, he needs to run his buildout command again to make it install
the eggs that are now specified for this latest release of Grok.

"Fixing up" current grokproject-made projects
---------------------------------------------

Projects that have been created with the current grokproject command
can be "fixed up" by:

1) removing the `newest = false` line from the project's
`buildout.cfg`.

2) adding the following two lines to the `[buildout]` section::

[buildout]
....
extends = http://grok.zope.org/releaseinfo/grok-0.10.1.cfg
versions = versions

and then running the buildout command again.

**NOTE**: this "fix up method" will work when Grok 0.10.1 has been
released. If you want to try anyway, you could try using this URL:
http://grok.zope.org/releaseinfo/grok-0.10.1b1.cfg


Next steps:

* Martijn and I have been working on a versions file with as much
final releases (instead of alphas and betas) in it as possible.
This needs to be finalized and tested on Windows. We can probably
do this on monday.

* Release 0.10.1 and people using Grok-0.10 should then be able
to use 0.10.1's versions list to "fix up" their projects.

* Review the jw-luciano-use-versions-file branch, and release
grokproject.

* Tweak the versions file for the grok trunk and release
Grok-0.11.


And of course, we're curious for you feedback!


kind regards,
jw

ps. I'd like to thank Luciano for pairing-up during the sprint!
It has been a pleasure to work together on this!