I have a scenario that contains Site's

I have a Site that contains Scenario's

The scenario1 is in Site[1].scenarios.

But the lookup "scenario in db.Site[1].scenarios" evaluates to false.

what does set(db.Site[1].scenarios) show?

I must have the wrong variable:

db.Scenario[1] in db.Site[1].scenarios

evaluates correctly.

what does set(db.Site[1].scenarios) show?

{Scenario[1]}

I think we're all good. I probably just need some lunch ?

:)

Is it possible that Site[1] and scenario on screenshot were from different db_session s?

I think we forgot to check this in __contains__

That is correct

It works only for objects from the same db_session

The scenario was from a previous db_session

Thank you. Next beer is on me.

Typically we forbid to mix objects from dfferent db sessions, but on this case we forgot to do the check

Sure :)

Hi. I have problem with memory. Script takes up 2Gb of RAM after creating 400k Book objects. Is there a way to disable caching for some classes or another solution? https://gist.github.com/Kurbezz/989a8cc82f56f203b6a61e239dbe6c6e

Not your main problem, but you can improve performance by doing author = Author.get(id=a_id)

and for your performance issue, how about batching your calls, so maybe doing 1000 "add" invocations per db_session

id_chunks = chunk(ids, 1000) for id_chunk in id_chunks: add(id_chunk)

code for chunk() https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks

Do you have yet another db_session, not displayed in gist code? Only the most outer db_session is taken into account, and inner db_sessions are ignored. The objects are clear out of memory on db_session exit. If you have yet another db_session which wraps all these code, the objects will stay in memory

Also, what version of Pony do you use?

Also, what version of Pony do you use?

0.7.3

Do you have yet another db_session, not displayed in gist code? Only the most outer db_session is taken into account, and inner db_sessions are ignored. The objects are clear out of memory on db_session exit. If you have yet another db_session which wraps all these code, the objects will stay in memory

updated gist

Can you enable SQL logging using set_sql_debug(True) in order to see where GET NEW CONNECTION and RELEASE CONNECTION messages are logged. Is it between each add function call, or just at the beginning and in the end of script execution?

In the updated gist code you have the following lines at the end of the script: temp_db.execute('DROP DATABASE temp;') temp_db.commit()

execute and commit can work inside db_session only. It means that you have some additional db_session which wraps all your script code and holds all the objects

You need to remove that outer db_session

Another (hackish) option is to add the following two lines at the end of add function: commit() rollback() commit() is necessary to save your objects to the database. Typically it is performed implicitly on the exit from the db_session rollback() forces clearing of all objects from the current db_session

Thanks

I just launched a test to postgres with 0.7.3. and got the error message:

File "C:\Anaconda3\envs\dce35\lib\site-packages\pony\orm\core.py", line 671, in _bind provider_module = import_module('pony.orm.dbproviders.' + provider) File "C:\Anaconda3\envs\dce35\lib\site-packages\pony\utils\utils.py", line 211, in import_module mod = __import__(name) File "C:\Anaconda3\envs\dce35\lib\site-packages\pony\orm\dbproviders\postgres.py", line 11, in <module> from psycopg2cffi import compat ImportError: No module named 'psycopg2cffi'

Is it plausble that some setup.py is missing a "required-packages" or a dot is missing in the import.

I don't know psycopg that well so my apologies if this is a stupid question.

If you use PostgreSQL , you need to install psycopg2. This is not included in requirements, because not all users use PostgreSQL

fair enough ?

Maybe we can improve the error message here

I use user = select(u for u in User if u.id_no == request.id_no and u.channel == channel and u.platform == request.platform).first() to get a user. And then do user.name='Halo', I didn't commit. after that, I check attr by if not user.id_card:. Is there a marked_to_delete action happen?

How are name and id_card related? Can you explain how deletion is involved?

IdCard has a ForeignKey relationship to User

And name just a field of User Entity

Ok, I'm still not sure what you're asking about deletion?

OK, I put it another way. When does the 'marked_to_deleted' status create

Looks like that's internal pony state, I think it's probably when you call user.delete() ?

Where are you seeing it?

There is a maked_to_delete , I am not sure when a entity marked as delete

in pony/orm/core.py it's set in Entity._delete_(), which is called by Entity.delete()

Looks like that's internal pony state, I think it's probably when you call user.delete() ?

So I think this is correct

Is there implicit delete action in Pony?

only for cascading deletes

but if you want to delete the user in your example, you need to call user.delete()

I pretty sure that I haven't called delete ,method

What behaviour are you witnessing?

It'd be helpful to show some code

and what is the problem you are having?

When I query the user's attr id_card, there seems to be a deletion to ContactInfo Entity. ContactInfo has a ForeignKey relationship to User

But I didn't call delete() method at all.

It's probably the case that it was either never created or it has been deferenced (no longer a foreign key relationship)

It's maybe.....Is that possible to check which Entity is been deleted by cehcking Pony internal vars?

I don't know

Ok, Thanks anyway :)

I suggest printing the object that you think has been deleted at multiple points in the code then you'll have a clear idea of what's happening

Neal, I think it is the following scenario: consider two entity classes A and B with one-to-one relationship: class A(db.Entity): id = PrimaryKey(int) b = Optional('B') class B(db.Entity): id = PrimaryKey(int) a = Required('A') Suppose you have two linked objects a1 and b1: a1 = A(id=1) b1 = B(id=1, a=a) assert a1.b is b1 assert b1.a is a1 Now you create new b2 object and link it with a1 in constructor: b2 = B(id=2, a=a1) In Pony all relationships are bi-directional: when you cnange one side of direction, the other side changes automatically at the same time. When you set b2.a to a1, Pony at the same time sets a1.b to b2, because the relationship is one-to-one, end each A object cannot be linked with more then one B object. But this operation unlinks a1.b from b1. Hence, we need to unset b1.a as well, it cannot have value a1 anymore. If attribute B.a where optional, we would be able to set b1.a to None. But B.a attribute is defined as Requried. So, the only things Pony can do in that situation is to delete b1, because it is the only way to satisfy requirements for b1.a. So, I think in your code you have the following classes: class User(db.Entity): contact_info = Optional('ContactInfo') ... class ContactInfo(db.Entity): user_id = Required('User') ... (by the way, with Pony the correct name for attribute is user, not user_id, because the attribute type is User, not int. If you want to name column user_id instead of user, it is possible to sprecify it in the attribute: user = Required('User', column='user_id')) You then execute the following query: old_contact = select(c for c in ContactInfo if c.owner_id == user and c.phone_no=contact.phone) With this query it is possible that user has contact, but it will not be selected, because the contact contains different phone_no value. In this cause you code will do the following: if not old_contact: ContactInfo(user_id=user, ...) As you defined the relationships between User and ContactInfo as one-to-one, that operation link user with new contact and need to delete old ContactInfo object, because the ContactInfo.user_id attribute is reqired and cannot be set to None. The root ot all problems is that you incorrectly defined the type of relationship: in User class instead of contact_info = Optional('Contact') you should have contacts = Set('Contact') These way you can have several different contacts for one user. The relationship type will be one-to-many (one user can have many contacts, one contacts belongs to one user), and adding new contact will not delete the previous contact

These are my Entity definition:

That's not a one-to-one relationship

I do have a table called jinlun_record in database and call_contact_id in that table is a ForeignKey references to primary key of ContactInfo Entity.

Ok, User.contact_list is indeed a Set, this is good. Then the problem is somewhere else. According to the traceback, the error is not directly related to line if not user.id_card, it just a coincidence that you see the error here. In order to retrieve user.id_card Pony needs to send a query to the database. But there ase some unsaved changes in code before that line. In order to be sure the query result is good, Pony issues flush() command to save previous changes. And during that save the error happens. You can manually perform flush() before that line, and this should lead to the same error. Then you can perform flush() earlier and find an update which lead to that problem. I'm still thinking that the problem caused by some incorrect relationshp. Something like what described in that StackOverflow answer: https://stackoverflow.com/questions/1905470/cannot-delete-or-update-a-parent-row-a-foreign-key-constraint-fails/1905480#1905480

new_id_card = IdCardPic(owner.id = User) user.id_card = new_id_card commit() I think the second statement is unnecessary.

yes

So maybe that caused the problem.

As I understand it, he encounter the problem earlier. I think the line user.id_card = new_id_card just does nothing, because user.id_card is already equal to new_id_card

Yeah right and actually the IntegrityError at the end tell exactly where is the problem.

In this traceback, obj is the user and id_card is the attr?

So what has been deleted is user?

https://gist.github.com/theresttime/4bb9e6436c603c4998fe9dbe269d7f8c

I can't reproduce the error in development environment. import_user_data was called by a thrift client.

if not user.id_card line has no direct relation to your problem. It just causes a saving of previously modified objects, which lead to this error. Some pevious code tries to delete ContactInfo, and it cannot be done because of record in jinlun_record table. I think problem happens outside of _write_profile_to_mysql function. I sugest you to insert flush() calls between previous lines of you code, and you will receive that error on one of that flush() call. Then you will know that the line before that flush() is the real source of the problem.

https://docs.ponyorm.com/working_with_entity_instances.html#updating-an-object Pony docs read Pony always saves the changes accumulated in the db_session() cache automatically before executing the following methods: select(), get(), exists(), execute() and commit(). So I think the old_contact = select(c for c in ContactInfo if c.owner_id == user and c.phone_no == contact.phone) part has already flushed the cache. Maybe the code block between select part and if not user.id_card produces the error? However, the code block neither changes the primary key of ContactInfo nor deletes the contactinfo....

Maybe profile.contact_list is empty, and the code don't execute that loop at all. I think adding flush() before if profile.contact_list and before if not user.id_card will make that clear

NO

I pretty sure it's not empty

In any case it will not harm to add that flush() commands here and there

It will make the root of the problem clear

That's error from production environment. I can't do that now :(

The problem looks pretty strange. The only situation when Pony may do implicit delete is with modification of one-to-one relationship, but in your case the relationship is one-to-many. I think it may be hard to solve the problem without some additional debugging. Deploying new version of code to production with additional flush() commands, would be pretty helpful to localising the problem. Also it will be helpful to enable pony SQL logging by calling sql_debug(True). Then you will see the query which causes the problem right before the traceback. Below the query you should be able to see query parameters, and it will give information which contactinfo object was mentioned. Also, looking at the traceback, it seems that you are using Pony version 0.7.1 at production. I think it is worth to upgrade to 0.7.3, because some bugs were fixed.