Grigory
probably, its a bug
Alexander
That's strange...
Grigory
It is fully reproducible, so I will try to make a minimum example
Alexander
That's great
Grigory
another funny detail is that if I use with db_session statement inside that function, the perfomance is normal
Grigory
so it is definitely something about the wrapper
Carel
I was reading the docs and saw mention of getting into an interactive session. It seems one can do so when invoking a module. Import code DB.bind(...) With db_session : Code.interact(locals=locals()) Will get me half way but when I query an entity it c
Alexander
Pony has a special mode which allows to not wrap a block of code with db_session. It may be useful when working in a console, because it is not possible to wrap multiple console commands with a single context manager, and typically a user want to have a single db_session when working in a console. So, when Pony is imported it tries to detect is it currently running from a console or not. If yes, Pony sets pony.MODE flag to "INTERACTIVE". After that each operation which requires db_session starts it implicitly if db_session wasn't started yet. This is what called "interactive mode" in Pony docs
Carel
Sorry I typed that like a dog having a scratch (I hit send instead of delete). I meant to say/ask if there is anything one should do to setup a script to drop into a interactive pony session. So if I want to interrogate the db while running a script as “python -m MODULE” can I use the code shown above as is or, given what you’ve said, should I do this : import code from pony.orm import * ... db.bind(...) ... pony.MODE =“INTERACTIVE” code.interactive(locals=locals()) Calling pdb.set_trace in a db_session seems to work alright for this.
Carel
Thanks for the epic answer though.
Alexander
I think the last code should work better, but after exiting from interactive mode you should restore the previous value of pony.MODE
Alexander
The former code should works too, but it will ignore db_session around functions that user called inside an interactive session, and it may be surprising behavior
Alexander
Pony interactive mode starts db_session implicitly, but it will not ignore explicitly started db_session
Carel
Ok, cool thanks ;)
Carel
It works as you say, which is kinda cool, now I can drop into an interactive shell in a Tornado app like one might in a Django one :)
Alexander
I think it will not work in Tornado this way
Carel
Oh, I’m doing this outside of the tornado eventloop. So querying the db by itself. I think if one does this with in the eventloop it’d be quite a different setup.
Alexander
By default Pony ties db_session to a current thread. It works well in single- or multi-threaded applications, where each thread represents idependent flow of work, but in an async application a single thread continually switches between different contexts. So, in async application Pony need to switch current db_session accordingly. To do that, Pony understand how to wrap db_aession around a generator function. When generator function yield a next result and suspends, Pony removes its db_session from the current thread, and reactivate it on entering the generator again. But it should be top-level db_session that wrapped the generator function
Carel
Wow, you really have all the bases covered. I’ll try an async function in a bit, just to try it out.
Alexander
Python's new async functions is a bit different than usual generators. We added it support recently, it is on GitHub, but not in the official released version yet. https://github.com/ponyorm/pony/issues/380 There was some comment that the support of async functions is not complete, but we was not able to reproduce the problem yet.
Lucky
Hey, I'm trying to create Entities without having to have a Database() already set up. So basically I use a different class (later = Later(), which just stores them in a list when using later.Entity), and at a later time try to get it into the database object: later.attach(db). I keep getting lost in those metaclass inherence problems. Any Idea how we could make this work? I'm thinking about something like flask's blueprints here. Here is my code so far: # -*- coding: utf-8 -*- from pony import orm from pony.orm import PrimaryKey, Required, Optional, Set from pony.orm.core import EntityMeta, Entity from luckydonaldUtils.logger import logging # from ...secrets import POSTGRES_HOST, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB __author__ = 'luckydonald' # __all__ = [] logger = logging.getLogger(__name__) class LaterMeta(EntityMeta): def __init__(self, name, bases, cls_dict): # self = later.Entity super(EntityMeta, self).__init__(name, bases, cls_dict) self._database_ = None # When created, check the LaterMeta class for the database for base_class in bases: if isinstance(base_class, LaterMeta): base_class._later.entities.append((self, name, bases, cls_dict)) # end if # end for # end def # end class class Later(object): """ :var Entity: LaterMeta """ def __init__(self): self._database_ = orm.Database() self.Entity = type.__new__(LaterMeta, 'Later', (Test,), {}) # type: LaterMeta self.Entity._later = self self.entities = list() def attach(self, database): class EntityMetaDatabased(EntityMeta): #def __new__(meta, name, bases, cls_dict): # return type.__new__(meta, name, bases, cls_dict) # pass __new__ = type.__new__ __init__ = object.__init__ #def __init__(*args): #type.__init__(name, bases, cls_dict) # pass e = EntityMetaDatabased() #class EntittyLater(database.Entity): # pass for table_args in self.entities: #entity = EntityMetaDatabased.__init__(*table_args) # lel = type.__new__(EntityMeta, 'Entity', (table_args[0].__class__,), {}) this = table_args[0] EntityMeta.__init__(this, table_args[0].__name__, (self,), {}) #database.Entity() # entity = type.__new__(EntittyLater, 'Entity', (), {}) # type: Entity pass later = Later() class Project(later.Entity): id = PrimaryKey(int, auto=True) proj = Required(str) # project identifier user = Required('User') default_lang = Optional(str) translations = Set('Translation') access = Set('Access') class User(later.Entity): id = PrimaryKey(int, auto=True) projects = Set(Project) translations = Set('Translation') access = Set('Access') states = Set('State') class Translation(later.Entity): id = PrimaryKey(int, auto=True) lang = Optional(str) project = Required(Project) key = Optional(str) # The programic key for the translation value = Optional(str) # The text user = Required(User) approved = Optional(bool) class Access(later.Entity): id = PrimaryKey(int, auto=True) project = Required(Project) user = Optional(User) # Either token or user token = Optional(str) # Either token or user class State(later.Entity): user = Required(User) chat_id = PrimaryKey(int) value = Optional(int) db = orm.Database() later.attach(db) db.bind(provider='sqlite', filename=":memory:", create_db=True) db.generate_mapping()
Lucky
So I want to get rid of def foo(db): class Bar(db.Entity) pass db = Database() foo(db) and replace it with something an IDE can inspect.
Lucky
Hey, I'm trying to create Entities without having to have a Database() already set up. So basically I use a different class (later = Later(), which just stores them in a list when using later.Entity), and at a later time try to get it into the database object: later.attach(db). I keep getting lost in those metaclass inherence problems. Any Idea how we could make this work? I'm thinking about something like flask's blueprints here. Here is my code so far: # -*- coding: utf-8 -*- from pony import orm from pony.orm import PrimaryKey, Required, Optional, Set from pony.orm.core import EntityMeta, Entity from luckydonaldUtils.logger import logging # from ...secrets import POSTGRES_HOST, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB __author__ = 'luckydonald' # __all__ = [] logger = logging.getLogger(__name__) class LaterMeta(EntityMeta): def __init__(self, name, bases, cls_dict): # self = later.Entity super(EntityMeta, self).__init__(name, bases, cls_dict) self._database_ = None # When created, check the LaterMeta class for the database for base_class in bases: if isinstance(base_class, LaterMeta): base_class._later.entities.append((self, name, bases, cls_dict)) # end if # end for # end def # end class class Later(object): """ :var Entity: LaterMeta """ def __init__(self): self._database_ = orm.Database() self.Entity = type.__new__(LaterMeta, 'Later', (Test,), {}) # type: LaterMeta self.Entity._later = self self.entities = list() def attach(self, database): class EntityMetaDatabased(EntityMeta): #def __new__(meta, name, bases, cls_dict): # return type.__new__(meta, name, bases, cls_dict) # pass __new__ = type.__new__ __init__ = object.__init__ #def __init__(*args): #type.__init__(name, bases, cls_dict) # pass e = EntityMetaDatabased() #class EntittyLater(database.Entity): # pass for table_args in self.entities: #entity = EntityMetaDatabased.__init__(*table_args) # lel = type.__new__(EntityMeta, 'Entity', (table_args[0].__class__,), {}) this = table_args[0] EntityMeta.__init__(this, table_args[0].__name__, (self,), {}) #database.Entity() # entity = type.__new__(EntittyLater, 'Entity', (), {}) # type: Entity pass later = Later() class Project(later.Entity): id = PrimaryKey(int, auto=True) proj = Required(str) # project identifier user = Required('User') default_lang = Optional(str) translations = Set('Translation') access = Set('Access') class User(later.Entity): id = PrimaryKey(int, auto=True) projects = Set(Project) translations = Set('Translation') access = Set('Access') states = Set('State') class Translation(later.Entity): id = PrimaryKey(int, auto=True) lang = Optional(str) project = Required(Project) key = Optional(str) # The programic key for the translation value = Optional(str) # The text user = Required(User) approved = Optional(bool) class Access(later.Entity): id = PrimaryKey(int, auto=True) project = Required(Project) user = Optional(User) # Either token or user token = Optional(str) # Either token or user class State(later.Entity): user = Required(User) chat_id = PrimaryKey(int) value = Optional(int) db = orm.Database() later.attach(db) db.bind(provider='sqlite', filename=":memory:", create_db=True) db.generate_mapping()
Alexander
Can you remind for what purpose do you use this pattern? ``` def foo(db): class Bar(db.Entity) pass db = Database() foo(db) ``` Is it to split entities to several modules?
Lucky
To be able to have them in another file, on in case of migrations (pony_up) load and unload them after migrating
Lucky
Basically i want to get those autocompletion goodness, but still have the models written down before needing to have a db connection
stsouko
`Try to def __dir__``
Lucky
Can you remind for what purpose do you use this pattern? ``` def foo(db): class Bar(db.Entity) pass db = Database() foo(db) ``` Is it to split entities to several modules?
Digging through it the problem so far is that there is no easy to use database.register(Entity) but all the checking and code is in the EntityMeta making that hard extend, use and so to postpone it until we have the database ready
Lucky
Basically I was hoping: Normal process: db.Entity calls self._database_.register_entity(self) Blueprint-is process: some sort of LaterEntity could just do self.register_queue.append(self), and have a function register_db(db): for e in self.register_queue: db.register_entity(e)
Lucky
So the goal would be like: from .models.something import Car db = Database(postgres="...") db.register_entity(Car)
Lucky
@metaprogrammer I had a thought about it. I think I can't get it to work without changes in pony. Basically the problem is that the metaclass already jumps into action, and several parts require a strict subclassing/instanceof which I can't really monkey patch
Lucky
Could maybe one of you guys look at it? That part of code isn't very well documented
Alexander
Hi @luckydonald, I'll think about it, may be we can change the process of entity class creation to make it more flexible. I cannot promise any specific at this point, as it may be a complex task
Alexander
> So the goal would be like: from .models.something import Car db = Database(postgres="...") db.register_entity(Car) I still don't understand in all details what is the use case, and why early-binding entity to database is not convenient in your case
Lucky
Yes, Basically it allows to easily switch databases on the fly, use the same models on different databases simultaneously and also to split those into several files, without circular imports (from model.foo import Bar which itself needs to import the other file from .. import db)
Lucky
What would be the Ponyorm way to do SELECT * FROM UserAccess WHERE user = ? AND project = ? to check if a given User u has UserAccess to a given Project p?
Lucky
What would be the Ponyorm way to do SELECT * FROM UserAccess WHERE user = ? AND project = ? to check if a given User u has UserAccess to a given Project p?
any(True for ua in UserAccess if ua.user == u and ua.project == p) is what I came up with, but not sure if that will work
Lucky
I have u = User.get(id=user_id) and p = Project.get(user=u, proj=project_key) and now need to know if they are connected with a UserAccess entity
Alexander
Hi @luckydonald! First, it is not necessary to have UserAccess entity in this diagram, because it is equivalent to many-to-many relationship. If you replace your current code with: class User(db.Entity): ... access_projects = Set("Project", reverse="access_users", table="useraccess", column="project") class Project(db.Entity): ... access_users = Set("User", reverse="access_projects", column="user") You even don't need a migration to work with previous tables The real need in separate UserAccess entity arises only when it has some additional attributes, like access_level or something like that Anyway, if you really need to have a separate UserAccess entity, you can use exists type of query: if not (p.access_all or UserAccess.exists(user=u, project=p)): return 403, {...} Also you can get all accessible projects for user using attribute lifting: user.access_projects.project will return a collection of all accessible projects to a specific user. Similar, project.access_users.user returns a collections of all users which can access a specific project. You can use attribute lifting in queries: select(p for p in Project if (p.access_all or u in p.access_users.user) and <some other conditions>)
Alexander
I don't understand what you mean by column="project". If you speak about UserAccess.exists(user=u, project=p) then yes, it will work even if project primary key is composite
Lucky
No, I ment your proposal for the class User(db.Entity):. Also you are right, I don't need a seperate table when using two Sets.
Alexander
I think yes, you can define additional many-to-many relationship between User and Project even if Project primary key includes reference to User But I recommend to use simple id primary key for Project, and define secondary unique composite key Project.name, Project.author
Alexander
At this moment diagram editior does not allow to define secondary composite keys, but you can add it manually after you take code of model definitions from the site
Lucky
So, basically I'd add composite_key(name, author) where I normaly would have PrimaryKey(name, author)?
Lucky
Is there a reason why one is CamelCase and one is snake_case?
Alexander
This is because PrimaryKey is an attribute class (like Required, Optional, etc.) which additionally can be used to define composite primary keys, and composite_index is just a function that registers additional indexes inside an entity. Maybe it would be more clear to have separate composite_pk function instead of (ab)using PrimaryKey attribute for composite primary keys
Lucky
I would name it primary_key(...), as there already is composite_key(...) but in general I agree.
Alexander
It was initially designed to be similar to SQL which has create table t1 ( foo int primary key, bar int ) as well as create table t2 ( foo int, bar int, primary key (foo, bar) ) so the same keyword name for single and composite primary keys
Lucky
yeah, I like that approach. I was just confusing shortly that they are different in capitalisation.
Lucky
Btw, I found a #bug in the editor.
Alexander
Ups, thanks for reporting
Lucky
Is there some kind of enum type, which would automatically fill the database? I'm thinking if I could model the Languages as a table, but having to insert those per hand wouldn't be fun...
Alexander
At this moment Pony does not support enums. You can define the following class class LANGUAGES: EN = 'EN' RU = 'RU' class MyEntity(db.Entity): lang = Required(str) ... MyEntity(lang=LANGUAGES.EN)
Lucky
Thanks
Alexander
Sure
Alexander
May be, I need to review the code. Often pool requests don't implement all necessary functionality (support of all databases, correct translation of queries to SQL, etc.) Hope I can look into it soon
Lucky
Coming ? https://github.com/ponyorm/pony/pull/392
That one just translates enums to an TEXT field. I was thinking of something like class Language(Enum): EN = 0 DE = 1 becoming something like: DROP TABLE IF EXISTS Language; CRATE TABLE Language id INTEGER PRIMARY KEY, value TEXT NOT NULL ; INSERT INTO Language ( id, value ) VALUES ( (0, 'EN'), (1, 'DE') );
Alexander
Some databases have native concept of enums. In my understanding this is what most users want when they says about enum support in Pony This is enums in PostgreSQL: https://www.postgresql.org/docs/11/static/datatype-enum.html And this is in MySQL: https://dev.mysql.com/doc/refman/8.0/en/enum.html But these enums are not totally equivalent to Python enums. There are many different ways to map Python enums to database enums, and it is hard to choose canonical mapping of Python enums to SQL enums. Some time ago I planned to add a special Enum type to Pony which is as close to database enum types as possible, but now with Python standard Enum class having a separate Pony Enum class looks a bit strange
Lucky
Hmm, python enums doesn't have to start with 0, they could just define FOOBAR = 4458
Alexander
You mean, with UserAccess entity?
Lucky
No, with your better version, the 2x Set
Lucky
Alexander
I think Project.get(name='foobar', owner=u) should work just fine
Alexander
Ah, you mean that the user is not owner, but just have an access
Lucky
hold on
Alexander
But in this diagram a user can have access to several projects with the same name, because only for owner the project name is unique
Lucky
no, indeed owner.
Lucky
I have the url /api/<owner_id:int>/<project_key>/<language>/<translation_key>, and want to check if that exists, and give an error if something doesn't. So basically check every part of the url for existence first.
Lucky
Lucky
Lucky
Alexander
Something like this: user = User.get(id=owner_id) if user is None: return 404 project = user.projects.select(lambda p: p.name == project_key).get() if project is None: return 404 original = project.originals.select(lambda o: o.name == language).get() if original is None: return 404 translation = original.translations.select(lambda t: t.lang == translation_key).get() if translation is None: return 404
Lucky
Uh yeah, that helps!
Lucky
Is that a technical limitation? I feel like having a .get(column=value) method similar to the classmethod would be quite handy