| Issue 5: | CCP import fails on '#System' entry | |
| 2 people starred this issue and may be notified of changes. | Back to list |
What steps will reproduce the problem?
Used the supplied instructions to download and import the latest ccp dump
The dump used: dom111-sqlite3-v1.db
What is the expected output? What do you see instead?
Expected the import to run through without errors, instead it failed when
linking some models, for example in invGroups and invTypes .
When importing invGroups, the row it failed on was:
(0, 0, u'#System', u'', None, 0, 1, 1, 0, 0, 0, 0)
I think the reason is that it tries to link it to a category '0' that does
not exist. To fix it I changed the following in class Importer_invGroups:
try:
category = InvCategory.objects.get(id=category_id)
except:
print 'invGroup Import Failed on', row
return
What version of the product are you using? On what operating system?
django-eve SVN, Django SVN, Linux, MySQL
Apr 26, 2010
Project Member
#1
dylan.liverman
Owner:
dylan.liverman
Apr 27, 2010
The output of the import script:
....
File
"/home/andre/projects/eve/django-eve/apps/eve_db/ccp_importer/importers/inventory.py", line
33, in import_row
category = InvCategory.objects.get(id=category_id)
....
eve_db.models.inventory.DoesNotExist: InvCategory matching query does not exist.
....
I'm using Python 2.5.2
I had a look and in the dump there is an invCategories with id=0, but it's not being
imported:
$ sqlite3 ccp_dump.db
SQLite version 3.4.2
Enter ".help" for instructions
sqlite> select * from invCategories where categoryID = 0;
0|#System|||0
sqlite>.exit
$ python manage.py eve_import_ccp_dump invCategories
Importing: invCategories
- invCategories: 100% [========================================] Time: 00:00:00
Import complete.
$ python manage.py shell
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from eve_db.models import *
>>> InvCategory.objects.get(id=0)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/andre/projects/eve/django-eve/django/db/models/manager.py", line 132,
in get
return self.get_query_set().get(*args, **kwargs)
File "/home/andre/projects/eve/django-eve/django/db/models/query.py", line 341, in get
% self.model._meta.object_name)
DoesNotExist: InvCategory matching query does not exist.
Not quite sure where the problem is yet, but it could be a Django/MySQL combination
with id=0. The value does get imported, but not with an ID of 0. In fact, each time I
re-run the import a new entry for #System gets created with a new ID:
>>> InvCategory.objects.count()
40
sqlite> select count(*) from invCategories;
29
$ python manage.py dbshell
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 11530
Server version: 5.0.51a-3ubuntu5.4-log (Ubuntu)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> select * from eve_db_invcategory where name = '#System';
+----+---------+-------------+--------------+------------+
| id | name | description | is_published | graphic_id |
+----+---------+-------------+--------------+------------+
| 42 | #System | | 0 | NULL |
| 44 | #System | | 0 | NULL |
| 46 | #System | | 0 | NULL |
| 48 | #System | | 0 | NULL |
| 50 | #System | | 0 | NULL |
| 52 | #System | | 0 | NULL |
| 54 | #System | | 0 | NULL |
+----+---------+-------------+--------------+------------+
7 rows in set (0.00 sec)
Apr 27, 2010
I believe I found the problem, it is MySQL related. Although now I'm not sure what the best solution would be. Does Django expect the Database to generate new values automatically? http://dev.mysql.com/doc/refman/5.0/en/server-sql-mode.html#sqlmode_no_auto_value_on_zero NO_AUTO_VALUE_ON_ZERO affects handling of AUTO_INCREMENT columns. Normally, you generate the next sequence number for the column by inserting either NULL or 0 into it. NO_AUTO_VALUE_ON_ZERO suppresses this behavior for 0 so that only NULL generates the next sequence number. This mode can be useful if 0 has been stored in a table's AUTO_INCREMENT column. (Storing 0 is not a recommended practice, by the way.) For example, if you dump the table with mysqldump and then reload it, MySQL normally generates new sequence numbers when it encounters the 0 values, resulting in a table with contents different from the one that was dumped. Enabling NO_AUTO_VALUE_ON_ZERO before reloading the dump file solves this problem. mysqldump now automatically includes in its output a statement that enables NO_AUTO_VALUE_ON_ZERO, to avoid this problem.
Apr 27, 2010
Looks like the best fix for this is going to be add an ID field as an IntegerField on every model where we have a CCP assigned PK. I can see more problems coming from this later if we don't. Every django model has an id = models.AutoField(primary_key=True) on it by default. This id is never addressed on tables like TypeReactions where there is no eve defined pk. On tables like invTypes where CCP gives us a TypeID, we just define the id but leave it as an AutoField. This works fine on PostgreSQL and SQLite.
Status:
Accepted
Apr 27, 2010
So if I understand correctly we don't directly map the Django id to the eve dump's id and just use it as a reference when needed. So the Django invcategory table would look like: id -- The Django auto id ccp_id -- The one in the ccp dump name description is_published graphic_id During import you then find the related objects by the ccp_id, but after that's done you won't usually need them.
Apr 27, 2010
We'd still use the CCP assigned ID. The id field can be overridden by assigning another field type to the id. You can even set any field to be a primary key. class invCategory: id = models.IntegerField(unique=True, primary_key=True) name = models.CharField(max_length=7) ... The primary key isn't restricted to be a number nor is it restricted to the name "id". For example: class foobar: name = models.CharField(max_length=7, unique=True, primary_key=True) Because of this django has an alias pk which always points to the primary key even if the field has been renamed. f = foobar.objects.get(pk="foobar") f = foobar.objects.get(name="foobar")
Apr 27, 2010
Great, I just tested it and it works. Index: inventory.py =================================================================== --- inventory.py (revision 44) +++ inventory.py (working copy) @@ -36,6 +36,7 @@ invCategories """ + id = models.IntegerField(unique=True, primary_key=True) name = models.CharField(max_length=50) description = models.CharField(max_length=255) is_published = models.BooleanField(default=True)
Apr 27, 2010
You are correct, it would need to be done for other models too. After fixing up invCategories there is now a problem with invGroups too.
Apr 28, 2010
I added the IntegerID fields to models in the inventory.py file. The issue should be resolved now.
Status:
Fixed
Apr 29, 2010
Thanks, I tested from a clean database and import and it ran through without any errors.
Apr 30, 2010
(No comment was entered for this change.)
Status:
Verified
|