My favorites | Sign in
Project Home
READ-ONLY: This project has been archived. For more information see this post.
Search
for
  Advanced search   Search tips   Subscriptions
Issue 5: CCP import fails on '#System' entry
2 people starred this issue and may be notified of changes. Back to list
Status:  Verified
Owner:  dylan.liverman
Closed:  Apr 2010


 
Project Member Reported by AndreFMi...@gmail.com, Apr 24, 2010
What steps will reproduce the problem?

Used the supplied instructions to download and import the latest ccp dump

The dump used: dom111-sqlite3-v1.db 

What is the expected output? What do you see instead?

Expected the import to run through without errors, instead it failed when
linking some models, for example in invGroups and invTypes .

When importing invGroups, the row it failed on was:

(0, 0, u'#System', u'', None, 0, 1, 1, 0, 0, 0, 0)

I think the reason is that it tries to link it to a category '0' that does
not exist. To fix it I changed the following in class Importer_invGroups:

    try:
        category = InvCategory.objects.get(id=category_id)
    except:
        print 'invGroup Import Failed on', row
        return

What version of the product are you using? On what operating system?

django-eve SVN, Django SVN, Linux, MySQL

Apr 26, 2010
Project Member #1 dylan.liverman
Took a look at this but I wasn't able to reproduce the error.  The dump has a 
category with id=0.  

(0, u'#System', u'', None, 0)

I'm using the Django SVN as well.  I don't believe it could be a MySQL problem.  
Could you post the output of the import script.  What is the import order?  What 
version of Python are you using?


Owner: dylan.liverman
Apr 27, 2010
Project Member #2 AndreFMi...@gmail.com
The output of the import script:

....
  File
"/home/andre/projects/eve/django-eve/apps/eve_db/ccp_importer/importers/inventory.py", line
33, in import_row
    category = InvCategory.objects.get(id=category_id)
.... 
eve_db.models.inventory.DoesNotExist: InvCategory matching query does not exist.
....

I'm using Python 2.5.2

I had a look and in the dump there is an invCategories with id=0, but it's not being
imported:

$ sqlite3 ccp_dump.db
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> select * from invCategories where categoryID = 0;
0|#System|||0
sqlite>.exit

$ python manage.py eve_import_ccp_dump invCategories
Importing: invCategories
 - invCategories: 100% [========================================] Time: 00:00:00
Import complete.

$ python manage.py shell
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from eve_db.models import *
>>> InvCategory.objects.get(id=0)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/andre/projects/eve/django-eve/django/db/models/manager.py", line 132,
in get
    return self.get_query_set().get(*args, **kwargs)
  File "/home/andre/projects/eve/django-eve/django/db/models/query.py", line 341, in get
    % self.model._meta.object_name)
DoesNotExist: InvCategory matching query does not exist.

Not quite sure where the problem is yet, but it could be a Django/MySQL combination
with id=0. The value does get imported, but not with an ID of 0. In fact, each time I
re-run the import a new entry for #System gets created with a new ID:

>>> InvCategory.objects.count()
40

sqlite> select count(*) from invCategories;
29

$ python manage.py dbshell
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11530
Server version: 5.0.51a-3ubuntu5.4-log (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> select * from eve_db_invcategory where name = '#System';
+----+---------+-------------+--------------+------------+
| id | name    | description | is_published | graphic_id |
+----+---------+-------------+--------------+------------+
| 42 | #System |             |            0 |       NULL |
| 44 | #System |             |            0 |       NULL |
| 46 | #System |             |            0 |       NULL |
| 48 | #System |             |            0 |       NULL |
| 50 | #System |             |            0 |       NULL |
| 52 | #System |             |            0 |       NULL |
| 54 | #System |             |            0 |       NULL |
+----+---------+-------------+--------------+------------+
7 rows in set (0.00 sec)



Apr 27, 2010
Project Member #3 AndreFMi...@gmail.com
I believe I found the problem, it is MySQL related. Although now I'm not sure what
the best solution would be. Does Django expect the Database to generate new values
automatically?

http://dev.mysql.com/doc/refman/5.0/en/server-sql-mode.html#sqlmode_no_auto_value_on_zero

NO_AUTO_VALUE_ON_ZERO affects handling of AUTO_INCREMENT columns. Normally, you
generate the next sequence number for the column by inserting either NULL or 0 into
it. NO_AUTO_VALUE_ON_ZERO suppresses this behavior for 0 so that only NULL generates
the next sequence number.

This mode can be useful if 0 has been stored in a table's AUTO_INCREMENT column.
(Storing 0 is not a recommended practice, by the way.) For example, if you dump the
table with mysqldump and then reload it, MySQL normally generates new sequence
numbers when it encounters the 0 values, resulting in a table with contents different
from the one that was dumped. Enabling NO_AUTO_VALUE_ON_ZERO before reloading the
dump file solves this problem. mysqldump now automatically includes in its output a
statement that enables NO_AUTO_VALUE_ON_ZERO, to avoid this problem. 
Apr 27, 2010
Project Member #4 dylan.liverman
Looks like the best fix for this is going to be add an ID field as an IntegerField on 
every model where we have a CCP assigned PK.  I can see more problems coming from 
this later if we don't.

Every django model has an id = models.AutoField(primary_key=True) on it by default.  
This id is never addressed on tables like TypeReactions where there is no eve defined 
pk.  On tables like invTypes where CCP gives us a TypeID, we just define the id but 
leave it as an AutoField.  This works fine on PostgreSQL and SQLite.


Status: Accepted
Apr 27, 2010
Project Member #5 AndreFMi...@gmail.com
So if I understand correctly we don't directly map the Django id to the eve dump's id
and just use it as a reference when needed.

So the Django invcategory table would look like:

id  -- The Django auto id
ccp_id -- The one in the ccp dump
name
description
is_published
graphic_id

During import you then find the related objects by the ccp_id, but after that's done
you won't usually need them.

Apr 27, 2010
Project Member #6 dylan.liverman
We'd still use the CCP assigned ID.  The id field can be overridden by assigning 
another field type to the id.  You can even set any field to be a primary key.

class invCategory:
id = models.IntegerField(unique=True, primary_key=True)
name = models.CharField(max_length=7)
...

The primary key isn't restricted to be a number nor is it restricted to the name 
"id".  For example:

class foobar:
name = models.CharField(max_length=7, unique=True, primary_key=True)

Because of this django has an alias pk which always points to the primary key even if 
the field has been renamed.

f = foobar.objects.get(pk="foobar")
f = foobar.objects.get(name="foobar")

Apr 27, 2010
Project Member #7 AndreFMi...@gmail.com
Great, I just tested it and it works.

Index: inventory.py
===================================================================
--- inventory.py        (revision 44)
+++ inventory.py        (working copy)
@@ -36,6 +36,7 @@

     invCategories
     """
+    id = models.IntegerField(unique=True, primary_key=True)
     name = models.CharField(max_length=50)
     description = models.CharField(max_length=255)
     is_published = models.BooleanField(default=True)

Apr 27, 2010
Project Member #8 AndreFMi...@gmail.com
You are correct, it would need to be done for other models too. After fixing up
invCategories there is now a problem with invGroups too.

Apr 28, 2010
Project Member #9 dylan.liverman
I added the IntegerID fields to models in the inventory.py file.  The issue should be 
resolved now.
Status: Fixed
Apr 29, 2010
Project Member #10 AndreFMi...@gmail.com
Thanks,

I tested from a clean database and import and it ran through without any errors.
Apr 30, 2010
Project Member #11 AndreFMi...@gmail.com
(No comment was entered for this change.)
Status: Verified

Powered by Google Project Hosting