I agree that we should leave the source code structure as it is currently done,
although with small changes (denoted with *):

 + sympy
 |
 | + core
   |
   | + tests (*)
 |
 | + modules
   |
   | + tests (*)
   | + polynomials
   | + specfun
   | + plotting
   | + geometry
   | + printing (*)
   | + concrete
   | + matrices (*)
   | + simplify (*)

I think it would be good to move mathml to printing module, as it is used only in
case of printing. I would also suggest to separate matrices.py and simplify.py into
smaller pieces in appropriate directories. It would be just easier to manage their
content.

In case of tests I'm not convinced if it is reasonable to mix them with sources.
Although it seems good idea to add 'tests' directories to core and modules. Then I
would leave internal structure in case of modules as it is ie. single test suit per
module. But in case of core tests I would rather create a test suit for some
particular functionalities rather than a source file. For example I would create
separate test_* files for assumptions, evaluation of powers (and others),
representations etc.

I think the tests in core should be in core/tests/ directory as you propose and
should be testing functionality. And in the modules I think they also should be in a
separate directory - and testing either for functionality, or files.

I agree completely. I think this would clean things up a bit, and be for the better.
Another change, as suggested in Issue 133, is to move pretty printing into modules.

core may need to use tools from modules. A direct
import of python modules will sooner or later lead
to cycling imports (I think all serious Python
programmers have had this kind of problems).

Therefore I suggest keeping a feature from sympy-research
that makes public symbols available to all modules
provided that the main package imports a module that
defines these public symbols. For example, according to 
sympy.core, all public symbols are attributes of the 
Basic class: Basic.Symbol, Basic.Rational, Basic.Limit, etc.
They become attributes at the moment pyhton imports the
the corresponding modules via using metaclass.

So, all modules just need to import Basic from basic.py
and have all symbols available via Basic class, no need
to import the corresponding modules from other modules.
In this way it doesn't really matter how the code is structured.

+1 for putting subpackage tests into sympy/package/subpackage/tests/,
this makes tests available also via sympy namespaces.

Why do we need intermediate modules subpackage? My suggestion:

+ sympy
 |
 | + core
   |
   | + tests (*)
 |
 | + polynomials
   |
   | + tests
 | + specfun
   |
   | + tests
 |
 etc.

Hmm, the nice thing about the modules directory is that it clearly describes that
"these things are modules" but other than that, I don't think there's any necessary
reason. I'd be happy with the structure proposed by Pearu or the same thing except
with a modules directory for all of the modules.

So it is redudant to have 'modules' (it is a bit like having
a tree sympy/subpackage/core etc..). Also, there will be a notion
conflict: in Python a module is a .py file, and what are under 'modules'
directory, are acctually subpackages. I think it's better not to redefine
notions that Python uses, that only would add more bits to documentation.

I myself prefer the modules directory, so that it clearly distinguishes the core
(that possibly can be rewritten to C/C++) and the rest. But that's just a cosmetic
issue - either is fine with me. So to decide - let's stick to modules for now, so
that the structure is the same as in the trunk and later we may remove the module
directory, if we want to.

So this issue is resolved, right? Let's start fixing it. The tests ( issue 192 ) will
be done using py.test. 

So first create the structure as described above, the same way as it is in trunk and
then start moving and fixing the modules togeter with tests. Ideally, all tests from
the trunk should be moved to the sympy-research (and possibly adapted). I am sure new
Issues will come up in the process.

I missed Pearu's post - what about the limits.py for example. They clearly belong to
the "modules", so they should be at the same level as "core" ? i.e. sympy/core/*,
sympy/limits.py ? what about tests for limits, should they go to
sympy/tests/test_limits.py?

I think limits.py and order.py (and also (o|)series methods) can be moved out
from the core provided that they can be accessed from core via some interface.

If we'll have scipy/limits.py then limits tests should be under sympy/tests/.

Could we make up a good name for a subpackage that contains limits, order,
and series tools? They are so closely related that should live together, I think.
May be sympy/series? Or sympy/asymptotics? Other suggestions?

I think sympy/series/ is fine. 

So I'm kind of lost as to the final decision. We'll stick to sympy/core and
sympy/modules. How about sympy/series, or will it actually be sympy/modules/series?

As soon as a decision is a little more clear, I can start to work on moving things
around right away. I'd like to get this one out of the way before starting work on
anything else (including fixing them) since this is only a very simple issue.

So if we are really sure about the structure so finally we should start the merge. I
would suggest to remove from new core all compatibility code because it does not help
at all (the core should be pure from now on). Also cleanup the main __init__.py. Then
please make the structure and copy only the modules which will work out of the box
(seems only series/limits and matrices, however in the other case it would be good to
separate the single source file into smaller pieces, for readability and
maintainability purpose). Other modules should be added as soon as they will be
compatible with new core (I don't mean they must use new features at this stage as
this can be done later).

Currently we have following modules:

 + series
 + matrices
 + plotting
 + polynomials
 + simplify
 + geometry
 + solvers
 + concrete ( includes sums_products.py)
 + printing ( includes mathml )
 + specfun
 + physics (*)
 + utilities (*) ( or just utils )

I've added two additional modules. One is 'physics', which would handle all stuff
needed by Ondrej, so for now it would include at least paulialgebra.py and maybe
sigma and gamma matrices. 

In case of 'utils' I'm not sure if it's worthy. Currently there is just single source
file for this. Whatever we will decide here please remember not to hide common code
like it was with any/all functions pair, somewhere deep in modules. Common
functionality should be exposed in 'utils' to all the developers (and well documented).

Do we use anywhere decorator.py and pkgdata.py? I've seen usage of the first in old
limits (but now it's not applicable). So if this is useless, should be removed from
source tree.

File trigonometric.py should be merged with defined_functions.py

OK. Let's stick to modules, i.e. this structure:

 + sympy
 |
 | + core
   |
   | + tests (*)
 |
 | + modules
   |
   | + tests (*)
   | + polynomials
   | + specfun
   | + plotting
   | + geometry
   | + printing (*)
   | + concrete
   | + matrices (*)
   | + simplify (*)

So that the structure is similar as it is now. When we move everything and fix
things, let's open the discussion again whether we should actually remove the module
directory.

Let's move the limits, series etc. into sympy/modules/series. Lets adapt the tests to
py.test. I agree with Mateusz above on all things. 

So first create the structure and then start fixing things. 

So who is responsible for making the structure?

btw, shouldn't we add to each source file licensing information?

Jason. But since you two want to work on it, why don't you exchange emails and ICQ,
so that you can coordinate things (like I am moving this file, you do this file)? 

Actually, if you know how to use IRC, that would be even better, since more people
can coordinate things. Let us know if you create some channel - we could put it on
our webpage.

(No comment was entered for this change.)

I've created the IRC channel at irc.freenode.net surprisingly named #sympy. There
will be also info in wiki on this.

Good idea. I'll be mostly unavailable for the next 4-5 hours, but after that I'll be
on there for sure to coordinate this.

In this case I think Mateusz can start working on this right now and Jason will join
in 5 hours.

New branch sympy-merge created with basic directory structure as discussed above. I
have also copied core sources but removed all compatibility code to make the new core
pure. I have also merged utils.py (as this was trivial). Next we should make a
schedule in which order merge modules and merge/rewrite/restructure core tests.

In case of modules I have tested each (more/less) and it seems they should be merged
as they are written in post #13. So, utilities is done, series is just a copy,
matrices works up to core failures. In case of polynomials I have pointed out to
Robert all (rather most) the failing places in his module. 

On simplify.py I will start work tomorrow. Geometry will work as soon as simplify()
will work. If matrices, simplify and polynmials will work so also solvers will work
(only ODE's may have to be changed as pattern matching has changed). Physics will
work when matrices will work (handling of left multiplication).

Printing is a topic for another issue. Currently core does not support any type of
pretty printing at all, so it here seems a lot of work to do. I think also with
specfun ther will be much work (even without religious discussion).

btw. please do your commits, in case of modules changes, only to trunk (as long as
the module isn't merged).

How about the decision on core tests: will it be per functionality or per file? I
have the files converted so that they will work with py.test. I can go ahead and
commit them, or if I should first break them up appropriately (if they need to be
broken up) then I can do that first. Once this is decided, I can commit everything
and  Issue 192  will be taken care of.

Commit it like you have it. We can restructuralize the tests later. My own opinion is
that in some cases it's better to test functionality and in some other cases it's
better to tests files. 

The tests to convert concern 'series' modules so copy it directly there. Anyway,
which files from core should be  moveed to this module: order.py, limit.py, ... ?

Is there a reason for the orthogonal_polynomials.py being in the core folder?

Also, can we add the module files to the merge-branch as soon as the tests work?

orthogonal_polynomials.py will be merged with 'specfun'. Of course we can add modules
sources but only those which work out of the box or which you currently work on (in
terms of merge) so that we will not mix new code with old.

Only order.py and limit.py.

Plus one must refactor oseries, _eval_oseries, _compute_oseries, as_leading_term,
_eval_as_leading_term, (ldegree, leadterm - the new core does not acctually use them,
we can remove them unless some other code is using them) methods from core classes. 

But I think copying and refactoring should be separate tasks, we need tests
to be in place before refactoring so that it would be easier to check that
nothing will break.

Expanding cos(n * x) uses Chebyshev from orthogonal_polynomials.py.
So, moving orthogonal_polynomials.py to modules adds modules
dependency to the core which is not good.
I think orthogonal_polynomials.py is in the same level of being or
not being in core as defined_functions.py and integer_sequences.py.

Thank you Pearu. I also agree that we should avoid changes to core unless it is
required to make modules work. We should concentrate on testing/documenting the core
and preparing extensive set of test suit. Later we will discuss what can be
removed/simplified/refactored.

The only thing I would change immediately are string representations, which are
currently quite explicit ie.:

(2*x+1)**2 --> 1 + 4 * x + 4 * x ** 2

I would rather expect it to output:

(2*x+1)**2 --> 1 + 4*x + 4*x**2

In case of orthopolys we will have time to discuss as 'specfun' module will be
probably done by the end of the merge.

Okay, I've committed the tests and also updated setup.py.

Actually, scrap that, I had problems committing. I have to catch my ride to school
for now, but I'll be back at it in 1 hour.

Okay, everything is committed now. I accidentally lost what I done first, so I had to
redo almost all of it again. I have also updated setup.py in the sympy-merge branch
so that the `test_core` and `test` options will work. Closing  issue 192 .

I've moved limit.py and order.py into sympy/modules/series and have updated some of
their tests so that most of them now pass. Overall, only 3 tests are now failing when
running `python setup.py test`.

Most things from this Issue were already implemented. Closing it. Let's create a new
Issue for each new problem that occurs.