Also, if my domain is example.com, I'd rather browse to example.com/admin than editor.ponyorm.com/username/project/admin

To sum it up, how I understood it, I don't think that is a practical idea, and nothing I would be comfortable using.

There's no way they plan it to be an external service

That would be ridiculous

We have the following scenario in mind: 1) A developer designs some ER-diagram 2) After that he has a database in the cloud which corresponds to that diagram 3) Then he can populate database via automatically generated admin interface, to be sure that the database is designed correctly 4) After that he get automatically generated backend API which allows him to work with this data programmatically 5) Then he can develop frontend/mobile application which speaks to that API 6) If he changes the diagram, the database migrates automatically and the API changes accordingly 7) He can download resulted backend application to his local server or continue to use it from the cloud

To sum it up, how I understood it, I don't think that is a practical idea, and nothing I would be comfortable using.

Thanks for sharing this. I think we should have two versions of the admin interface. One which works locally, and another one for working with the database which is depolyed in the colud

We have the following scenario in mind: 1) A developer designs some ER-diagram 2) After that he has a database in the cloud which corresponds to that diagram 3) Then he can populate database via automatically generated admin interface, to be sure that the database is designed correctly 4) After that he get automatically generated backend API which allows him to work with this data programmatically 5) Then he can develop frontend/mobile application which speaks to that API 6) If he changes the diagram, the database migrates automatically and the API changes accordingly 7) He can download resulted backend application to his local server or continue to use it from the cloud

So pony orm will phone back to your servers when ever I change something? Shortly said: Eww!

No, why phone back? I think we speak about different things

Thanks for sharing this. I think we should have two versions of the admin interface. One which works locally, and another one for working with the database which is depolyed in the colud

This would be great. Also deploying a service while still having it open source could work too. Take for instance sentry.io. Their complete infrastructure is open source. This means poor students like me can deploy it on the're own server (they got for 22 bucks on ebay) in the basement. Means I'll already use it regularly and maybe later will use it in my job, too. Paid of course.

Probably not

Bad example, you probably want to run your own instance of Sentry anyway

We do, at least

Sure, you will get a full application which can be deployed on your own server and used without any cloud service. The cloud backend is just an option for those developers who want to have "dumb" backend for their mobile applications, and don't want to mess with administering a server

Yeah that's reasonable

Bad example, you probably want to run your own instance of Sentry anyway

I do too xD Because I can't affort it

+

Sure, you will get a full application which can be deployed on your own server and used without any cloud service. The cloud backend is just an option for those developers who want to have "dumb" backend for their mobile applications, and don't want to mess with administering a server

Oh, thats cool

I was mistaken than, sorry.

I have this code: http://pastebin.com/0ztWZmRu (just pseudo code) And when 100 threads writing some data i got IntegrityError (cause obj A or C can be written in different threads). Then i add to db_session retry=1. And now i sometimes got error like "mix objects from different transactions". I have a 2 questions: 1. What a right way to concurrently upsert objects? 2. How correctly retry writing on TransactionError (IntegrityError)?

Hi Святослав! 1) We need to add support of native PostgreSQL insert ... on conflict update .... Until that, retry option should works. But retry=1 is not enough, because once in a while the same error can occure again. I'd use retry=5 or retry=10. 2) Nested db_sessions are ignored, only the topmost one is taken into account. You need to apply @db_session to a function which wraps the entire atomic transaction, like, handling of HTTP request or loading a next batch of objects. The write method in your example is just a part of transaction, and donsn't look like a good candidate for @db_session.

I'd just like to point out how much I appreciate that you guys avoid talking in slavic languages here

Totally agree. This is fantastic!

👋

Hi Святослав! 1) We need to add support of native PostgreSQL insert ... on conflict update .... Until that, retry option should works. But retry=1 is not enough, because once in a while the same error can occure again. I'd use retry=5 or retry=10. 2) Nested db_sessions are ignored, only the topmost one is taken into account. You need to apply @db_session to a function which wraps the entire atomic transaction, like, handling of HTTP request or loading a next batch of objects. The write method in your example is just a part of transaction, and donsn't look like a good candidate for @db_session.

Hmm. If nested db_session ignored, then i have only one transaction. Then what mean error about "mixin" object from different transactions? How it happens? Internal caches + multithreading?

I mean each thread start transaction, and only one way to collide with different thread is Pony internals. Cause after my write method objects not reused

PS: My previous point to "obj A or C can be written in different threads" is incorrect. Object and their depends will be written in the same thread.

It can happen in the following situation: # no outer db-session obj1 = a.write() # first db_session # first db_session is over b.write(a=obj1) # second db_session I suspect something like that happened. You call several write methods in a line, and pass pony objects created in a former db_session to a latter one

Instead, you need to wrap such a sequence in a single db_session

Ok! Thanks!

I will check my code for this case.

Is there a post commit hook in pony orm? So I can execute a Python function every time pony updates a object (row) on e.g. `Person` table?

Basically I want to notify that row to my celery working queue, so that a worker later would update/crate a fitting representation in elastic search

You can define entity methods after_insert, after_update and after_delete: class MyPerson(db.Entity): name = Required(str) def after_update(self): print('Person %s was updated' % self.name)

Awesome!

These methods are called after insert/update/delete, but before commit, and can contains some cascade actions

Is there something after the commit?

At this moment - no. If you use db_session as a decorator, you can use custom decorator which call hook after exiting from db_session. Something like that (not tested): def db_session_with_post_commit_hook(hook): def decorator(func): def new_func(*args, **kwargs): with db_session: func(*args, **kwargs): hook() return new_func return decorator def print_msg(): print('after commit!') @db_session_with_post_commit_hook(print_msg) def my_function(): select(obj for obj in MyObject)

<class 'tuple'>: ('Object Category[new:1] cannot be stored in the database. InternalError: ОШИБКА: текущая транзакция прервана, команды до конца блока транзакции игнорируются\n',)

http://pastebin.com/iWyDQRdf

I got this error on obj = self._get_or_create(parent=parent, **category)

each retry failed cause "transaction interrupted"

In some places i do manually commit to get pk after object creating, can this be reason for error above?

I don't think so

As I see, you apply @db_session to operations like _get_or_create it seems that @db_session should be defined on some outer function instead, because _get_or_create is just a part of transaction.

It's defined outer too

or you mean it should be single definition?

Not necessary, but nested db_session will be ignored with their retry options

it's failed on second object, when first will be written as new and reused as parent for next

What is primary key of Category?

class Category(db.Entity): id = orm.PrimaryKey(int, auto=True) en_name = orm.Required(str) ru_name = orm.Optional(str, nullable=True) ads = orm.Set(Advertiesment) categories = orm.Set('Category', reverse='parent') parent = orm.Optional('Category', reverse='categories', column='parent_id') orm.composite_key(en_name, parent)

Maybe you are putting to much into a single transaction. When you insert an object, it remains lock until the end of transaction, and if you do some slow operations after inserting an object you can get conflicts when another transaction tries to insert an object with the same unique key. If you use operation like get_or_create, it should be put in a short independent transaction, without updating all other stuff.

You may try to use raw sql query to call native INSERT ... ON CONFLICT DO NOTHING, maybe it will perform better

Previously I think db_session start new transaction. I can do this explicit?

It does

https://docs.ponyorm.com/transactions.html

OK, thank you for explanation

So, when commit new object to db, this cause flush db_session cache. And when i start next transaction, using previous objects causing `pony.orm.core.TransactionError("An attempt to mix objects belonging to different transactions") What am I doing wrong? It seems I use the wrong tools

You are right, exiting from db_session causes implicit commit and invalidates all used objects so they can be garbage collected

In my simple example i have list of categories. Where each category is parent for next. First has parent=None

Which idiomatic way to resolve such troubles? In other words: how to create object and use it in the same transaction.

"create" mean "get or create"

In typical situation in Pony you can create object and use it in the same transacton without any problem with db_session: obj = MyEntity(**kwargs) # or MyEntity.get(**kwargs) # do something with obj flush() # or commit() # do something else with obj # end of db_session - implicit commit Using 'get or create' is tricky, because you start encounter concurrency issues, when concurrent transactions insert the same object. In general it is better not to use 'get or create' pattern, especially in high-concurrent environment. Maybe you can restructure API in such a way that creating a Category happens as a separate atomic action, independent of any other actions.

https://www.depesz.com/2012/06/10/why-is-upsert-so-complicated/

If I understand your use case correctly, you have a number of items which belongs to different hierarchical categories. I think it is better to represent such hierarchical categories as a text strings. For example, if you have some item placed in category "foo" -> "bar" -> "baz", you can have category text attribute with value 'foo|bar|baz'. If you make this attribute indexed, you can quickly find all items which belong to the same category. Another similar attribute may be for russian category names. Then if you add new item, you will not have any concurrency issues. A path into category hierarchy will be just a text column stored for that item. To search items belonging to some subcategory "foo|bar|baz" you just need to find all items which category path is between "foo|bar|baz" and "foo|bar|baz|zzzzzzzzzz". In my opinion such approach looks more manageable

Interesting idea

Hey team! How's migration going? Was looking to change a 0,1---0,1 relationship to 0,n---0,1

Need to have a Set on the user side, to allow multible logins across different browsers.

Hi @luckydonald! The migration tool is almost ready, we are fixing last bugs (I don't want to receive a flood of requests "the migration tool broke my database, please restore my data back") In your very specific case you don't need any migration at all — just replace web_session = Optional("WebSession") to web_sessions = Set("WebSession"), and new program will be able to continue working with the old database

I am driving right now. I will get back to you soon.

I have a slightly *strange* issue... I'm trying to extend the flask-security plugin to allow for Pony support (since I use it throughout my project, and I don't want to switch over to sqlalchemy). In a nutshell, they require a many-to-many relationship between a User and a Role class, however they also assume that to add a new entry to that Set(), the syntax is append. Pony uses add. Is there an easy/quick way to set up an alias from append to add?

You can do the following hack: from pony.orm.core import SetInstance SetInstance.append = SetInstance.add

Ahhh, nifty, let me try that....

Does the order of importing Set matter? Should I run that before I import set? or after? or does it not affect?

It doesn't matter

That worked great! Thanks!

Hi @luckydonald! The migration tool is almost ready, we are fixing last bugs (I don't want to receive a flood of requests "the migration tool broke my database, please restore my data back") In your very specific case you don't need any migration at all — just replace web_session = Optional("WebSession") to web_sessions = Set("WebSession"), and new program will be able to continue working with the old database

Now I am getting pony.orm.dbapiprovider.ProgrammingError: column "user" does not exist I assume on the WebSession table (the error could be more specific).

Now I am getting pony.orm.dbapiprovider.ProgrammingError: column "user" does not exist I assume on the WebSession table (the error could be more specific).

Indeed there was no such table generated.

Just to cover that case in migrations

I now have class User(db.Entity): # ... web_session = Optional('WebSession') # TODO: remove later. Kept for compatibility. web_sessions = Set('WebSession', reverse="user") And class WebSession(db.Entity): # ... user = Optional(User)