Jacob
Is this a safe way to "unbind" a Database() instance?
db.provider = db.provider_name = db.schema = None
Alexander
No, there is no way to unbind database at this moment, but you can define entities inside a factory function
def define_entities(db):
class Foo(db.entity):
a = Required(int)
b = Required(str)
PrimaryKey(a, b)
bars = Set("Bar")
class Bar(db.Entity):
foo = Required("Foo")
x = Required(int)
return Foo, Bar
db1 = Database()
db1.bind(**db_params)
Foo, Bar = dfine_entities(db1)
with db_session:
q1 = select(f for f in Foo)
q2 = select(b for b in db.Bar)
db1.disconnect()
db2 = Database()
db2.bind(**db_params)
define_entities(db2)
Jacob
OK. I will go that route. (I second the request for declarative model blueprints)
Jacob
Is there anything I or anyone else in the community can help with to get closer to releasing migrations? (write code, review code or api, run tests, write tests, use hypothesis to expand tests, write or edit docs, pre-alpha-test the editor...)
Matthew
I have a weird bug:
Matthew
print('campaigns counter:', counter)
all_campaign_facebook_ids.add(campaign_data['id'])
facebook_campaign = models.FacebookCampaign.get(lambda c: c.facebook_ad_account == facebook_ad_account and
c.facebook_id == campaign_data['id'])
print(facebook_campaign)
Matthew
On the second iteration of a for loop, that code "locks up" between the two print statements
Matthew
this bug is happening consistently
Matthew
any idea how I can debug what's going on? I guess it's possible that the database driver is never returning data?
Matthew
(postgres)
Matthew
There is no async or threading involved
Alexander
Matthew, I suggest you to set_sql_debug(True) and check what is the last SQL statement.
Probably some other transaction (in different thread or in different process) hold some lock which halts current transaction's SELECT operation. For example, other transaction can do SELECT FOR UPDATE or ALTER TABLE
Matthew
Thank you, I will try that
Alexander
Jacob, thanks for the help suggestion, maybe we contact to you later for help with code review
Jacob
Jacob
This is what I came up with to use the factory method for models in a way that feels a bit cleaner.
from pony.orm import *
# noinspection PyUnusedLocal
def attach_models(db):
class Thing(db.Entity):
_table_ = "awfullegacythingstablename"
id = PrimaryKey(
int,
column="AwfulInconsistentIDFieldName",
auto=True)
# ... other fields
class LegacyDatabase(Database):
# noinspection PyMethodParameters
def generate_mapping(
database,
filename=None,
check_tables=True,
create_tables=False
):
assert not create_tables, (
"Do not use PonyORM to create tables "
"in LegacyDatabase. The database MUST "
"already be up with schema loaded."
)
attach_models(database)
super(
LegacyDatabase,
database
).generate_mapping(
filename=filename,
check_tables=check_tables,
create_tables=False
)
db_params = {
# ...
}
db = LegacyDatabase()
db.bind(**db_params)
db.generate_mapping()
with db_session:
things = select(t for t in db.Thing)
Will I regret subclassing Database like this? (Does it cause known issues?)
Matthew
Matthew, I suggest you to set_sql_debug(True) and check what is the last SQL statement.
Probably some other transaction (in different thread or in different process) hold some lock which halts current transaction's SELECT operation. For example, other transaction can do SELECT FOR UPDATE or ALTER TABLE
It seems like it was stuck on a lock, probably an old python process holding the lock. A reboot seems to have fixed it :)
Alexander
This is what I came up with to use the factory method for models in a way that feels a bit cleaner.
from pony.orm import *
# noinspection PyUnusedLocal
def attach_models(db):
class Thing(db.Entity):
_table_ = "awfullegacythingstablename"
id = PrimaryKey(
int,
column="AwfulInconsistentIDFieldName",
auto=True)
# ... other fields
class LegacyDatabase(Database):
# noinspection PyMethodParameters
def generate_mapping(
database,
filename=None,
check_tables=True,
create_tables=False
):
assert not create_tables, (
"Do not use PonyORM to create tables "
"in LegacyDatabase. The database MUST "
"already be up with schema loaded."
)
attach_models(database)
super(
LegacyDatabase,
database
).generate_mapping(
filename=filename,
check_tables=check_tables,
create_tables=False
)
db_params = {
# ...
}
db = LegacyDatabase()
db.bind(**db_params)
db.generate_mapping()
with db_session:
things = select(t for t in db.Thing)
Will I regret subclassing Database like this? (Does it cause known issues?)
I think it should work. Anyway in case of problem it can be easily refactored to separate function applied to db before generate_mapping
Lucky
Is there anything I or anyone else in the community can help with to get closer to releasing migrations? (write code, review code or api, run tests, write tests, use hypothesis to expand tests, write or edit docs, pre-alpha-test the editor...)
I second that, that would be really needed to moved forward
Lucky
This is what I came up with to use the factory method for models in a way that feels a bit cleaner.
from pony.orm import *
# noinspection PyUnusedLocal
def attach_models(db):
class Thing(db.Entity):
_table_ = "awfullegacythingstablename"
id = PrimaryKey(
int,
column="AwfulInconsistentIDFieldName",
auto=True)
# ... other fields
class LegacyDatabase(Database):
# noinspection PyMethodParameters
def generate_mapping(
database,
filename=None,
check_tables=True,
create_tables=False
):
assert not create_tables, (
"Do not use PonyORM to create tables "
"in LegacyDatabase. The database MUST "
"already be up with schema loaded."
)
attach_models(database)
super(
LegacyDatabase,
database
).generate_mapping(
filename=filename,
check_tables=check_tables,
create_tables=False
)
db_params = {
# ...
}
db = LegacyDatabase()
db.bind(**db_params)
db.generate_mapping()
with db_session:
things = select(t for t in db.Thing)
Will I regret subclassing Database like this? (Does it cause known issues?)
I would go with the standart self argument though, that is more clear that it is indeed the current instance.
Lucky
For now I prefer the manual "save" button, so I don't accidentially f**k up my diagram.
Often I just drag stuff arround, and position isn't undo-able it seems.
Anyway, people love usecase descriptions, so here 'ya go:
I often add stuff, but am not happy with it, and just don't save the changes I made.
Jacob
I would go with the standart self argument though, that is more clear that it is indeed the current instance.
lol. In this case, I'm copying the way pony does it internallly (and probably always will[1][2]) so that it is easier for pony core devs to comment :)
[1] https://github.com/ponyorm/pony/issues/184#issuecomment-256015841
[2] https://github.com/ponyorm/pony/issues/184#issuecomment-352122658
Lucky
Honestly, I think it's horrible
Lucky
A bit more PEP8 would be really nice.
Lucky
I wouldn't mind self_database though. That feels like a good compromise
Jacob
It looks like, after migrations comes out (so that no one has to deal with merge hell with private branches), we can get pep8 everywhere except for self/cls.
Jacob
At least, that's my hope.
Lucky
In the pony orm code it's really hard to see if a variable is the instance or just, you know, a variable
Jacob
/me shrugs
Jacob
/me needs to learn all of the telegram formatting - but irc habits don't die easily
Lucky
I miss @bonbot in this chat
Lucky
That could do /me
Lucky
luckydonald would really like that here
Lucky
channels = Channel.select(
lambda c: (
isinstance(c, Channel) and
(current_time_in_minutes - c.posting_offset) % c.posting_interval == 0
)
)
results in
TypeError: Function 'isinstance' cannot be used this way: isinstance(c, Channel)
It would be quite helpful to get the IDE to recognize the c variable as instance of type Channel.
Alexander
I expected isinstance should work here. I can look into it a bit later
Lucky
Why does query.get_sql() needs a db_session? It just shuld generate SQL.
Martin
Yes, if transaction wasn't already stared a new transaction will be started automatically before db.get_connection() returns connection
When is a transaction started, is it at the creation of db object or at the execution of db.execute()
Martin
I am rewriting few legacy code with pony orm but using pony for connection pooling and still using legacy queries and procedures...
just want to make sure, is transaction management handled properly by my code
Martin
class DataAccess:
"""
API implementation for a relation database access.
"""
@db_session
def select(self, query, query_args=None, classname=None, output_as_dict=False):
"""
select() method uses the query string provided for retrieving information
:param query: Query to be executed. Parameter to be passed had to be prefixed with $ sign.
e.g: "select id, name from Person where name = $x"
:param classname:
:param query_args: Arguments that need to be passed to the query. e.g: {"x" : "Susan"}
:param output_as_dict: When true, output will be returned as list of dict objects :e.g: [{"x" : "Susan"}],
if below condition is satisfied, classname is None and "*" is not provided in select
query for columns
:return:
"""
data = self.db.select(query, query_args)
# if classname or output as dict is true
# creation of key values is done
# else list of tuples is returned
if classname or output_as_dict:
column_names = query.split('select ', 1)[1].split('from')[0].strip()
# if * is provided in the column names list
# return the tuple list
if column_names.find("*") == -1:
return data
column_names_list = column_names.split(',')
data_as_dict_list = []
# loop over list of tuples
# create a dict item
# with column names from select list by index and
# value as tuple element by index
for item in data:
dict_item = {}
for idx, elem in enumerate(item):
dict_item.update({column_names_list[idx].strip(): elem})
data_as_dict_list.append(dict_item)
if classname:
data_class_list = []
for idx, dict_item in enumerate(data_as_dict_list):
data_as_dict_list.append(namedtuple(classname, dict_item.keys())(*dict_item.values()))
return data_class_list
else:
return data_as_dict_list
else:
return data
@db_session
def create(self, query, callback=None, query_args=None):
"""
create() method takes the query string for creating table
:param query:
:param callback:
:param query_args: Arguments that need to be passed to the query. e.g: {"x" : "Susan"}
:return:
"""
data = self.db.execute(query, query_args)
return data
@db_session
def insert(self, query, callback=None, query_args=None):
"""
insert() method for inserting an item
:param query:
:param callback:
:param query_args:
:return:
"""
data = self.db.execute(query, query_args)
return data
@db_session
def update(self, query, callback=None, query_args=None):
"""
update() method for updating information of a given item
:param query:
:param callback:
:param query_args:
:return:
"""
data = self.db.execute(query, query_args)
return data
@db_session
def delete(self, query, callback=None, query_args=None):
"""
delete() method for deleting an item
:param query:
:param callback:
:param query_args:
:return:
"""
data = self.db.execute(query, query_args)
return data
@db_session
def execute(self, query, classname=None, query_args=None):
"""
execute() method for executing a procedure/function
:param query:
:param classname:
:param query_args:
:return:
"""
data = self.db.execute(query, query_args)
return data
@db_session
def execute_procedure(self, procedure_name_params, procedure_args=None):
"""
execute_procedure() method for executing a procedure
:param
Martin
procedure_name_params: name of a procedure and if any IN params e.g: myprocedure($x, $y).
ALL IN parameters had to be prefixed with $ symbol
:param procedure_args: any parameters that had to be passed to procedure {'x': 10, 'y': 20}
:return:
"""
data = self.db.execute("call " + procedure_name_params, procedure_args)
return data
@db_session
def execute_function(self, function_name_params, function_args=None):
"""
execute_function() method for executing a function
:param function_name_params: name of a function and if any IN params e.g: myfunction($x, $y).
ALL IN parameters had to be prefixed with $ symbol
:param function_args: any parameters that had to be passed to function {'x': 10, 'y': 20}
:return:
"""
data = self.db.execute("select " + function_name_params, function_args)
return data
@db_session
def begin_transaction(self):
"""
begin_transaction() method for initiating transaction
:return:
"""
self.db.commit(False)
@db_session
def end_transaction(self):
"""
end_transaction() method for end a transaction
:return:
"""
self.db.commit(True)
@db_session
def rollback(self):
"""
rollback() rollback a transaction
:return:
"""
self.db.rollback()
def __init__(self, provider, hostname, **kwargs):
"""
Initializes FTP, SFTP setup
:param provider: [MySQL, Oracle, PostgreSQL] only these providers are supported as of now
:param hostname:
:param kwargs:
username - username to access database
password - password of the user
db - database instance
"""
self.provider = provider
self.hostname = hostname
self.username = kwargs.get('username')
self.password = kwargs.get('password')
self.database = kwargs.get('database')
if self.provider not in SupportedDatabase.__members__:
db_list = ','.join(list(SupportedDatabase.__members__))
raise Exception(
provider + ', is not supported at this time. Following databases are only supported : ' + db_list)
self.db = Database()
self.db.bind(provider=SupportedDatabase[self.provider].value, user=self.username, password=self.password,
host=self.hostname, database=self.database)
Lucky
Hey, I'm getting an exception in the pony ORM core:
Traceback (most recent call last):
File "/app/src/teleflask-c340ab/teleflask/server/mixins.py", line 172, in process_update
self.process_result(update, listener(update)) # this will be TeleflaskMixinBase.process_result()
File "/app/src/telestate-a6199f/telestate/machine.py", line 299, in process_update
self.save_state_for_chat_user(chat_id, user_id, state_name, state_data)
File "<string>", line 2, in save_state_for_chat_user
File "/usr/local/lib/python3.6/site-packages/pony/orm/core.py", line 528, in new_func
result = func(*args, **kwargs)
File "/app/src/telestate-a6199f/telestate/contrib/pony.py", line 120, in save_state_for_chat_user
data=state_data,
File "<string>", line 2, in init
File "/usr/local/lib/python3.6/site-packages/pony/utils/utils.py", line 59, in cut_traceback
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pony/orm/core.py", line 4673, in init
entity._get_from_identity_map_(pkval, 'created', undo_funcs=undo_funcs, obj_to_init=obj)
File "/usr/local/lib/python3.6/site-packages/pony/orm/core.py", line 4378, in _get_from_identity_map_
if obj._pk_is_composite_: pairs = izip(pk_attrs, pkval)
TypeError: zip argument #2 must support iteration
Permalink Bot
If you wanna have a look at that mentioned save_state_for_chat_user, that's on on github
Permanent link to the file telestate/contrib/pony.py mentioned. (?)
Alexander
Thanks for reporting, I cannot look at code right now, will fix it a bit later
Lucky
Also found a bug in the editor again #editor #bug
Lucky
The difference between orm.Optional and orm.Required(..., nullable=True) is that
- Required means you have to provide it in python code
when creating a new entity, like foo = Foo(field=None),
- while Optional means you don't have to use it and it uses default=... or sql_default="...", right?
Alexander
>>> class Test(db.Entity):
... attr = Required(int, nullable=True)
...
>>> db.generate_mapping(create_tables=True)
>>> Test(attr=None)
Will give you ValueError: Attribute Test.attr is required
Lucky
user_id = orm.Optional(int, sql_default='NULL', default=None, index=True, nullable=True) # can be None (e.g. channels)
TypeError: Default value for non-nullable attribute State.user_id cannot be set to None
Matthew
Try default=0
Matthew
If that doesn’t make sense, don’t specify a default?
Martin
>>> class Test(db.Entity):
... attr = Required(int, nullable=True)
...
>>> db.generate_mapping(create_tables=True)
>>> Test(attr=None)
Will give you ValueError: Attribute Test.attr is required
how to paste the code, which is formatted like this
Juan Antonio
Yo need to use triple quotes like this code
Juan Antonio
Lucky
code lol
Rozen
the image xD
Rozen
well, the youtube thing xD
Lucky
https://www.youtube.com/watch?v=8yu9j225q-8
Lucky
Littlepip is best pony. Sorry, PonyORM.
—
This is offtopic however, therefore I'll now shut up :D
Martin
Lucky
How do I use select with the same style like get: Entity.select(foo="bar")?
Lucky
Deleted last message, it's more difficult then I thought.
Lucky
Post.get(channel=self, reddit_id=submission.id) is None
should be the same as
Post.select().filter(channel=self, reddit_id=submission.id).exists()
Is it last one faster, as it doesn't need to download/pythonify an object?
Lucky
Also is there a way to get the result of a raw sql query in something text-based?
I need a quick way to query the db and print the result as text.
Alexey
Lucky
It looks like, after migrations comes out (so that no one has to deal with merge hell with private branches), we can get pep8 everywhere except for self/cls.
I also think we could split the core into more files, like one for Set, one for Query, etc.
Lucky
#editor #bug Editing snapshot names causes it to go all white.
Matthew
pony.orm.core.UnrepeatableReadError: Value of FacebookAdAccount.last_updated for FacebookAdAccount[5] was updated outside of current transaction
Matthew
Does this mean that another session has updated that value? Can it happen any other way?
Alexander
This error means that the field values was changed outside of the ORM code. It is possible that the field was updated by a trigger or a raw sql query. In that case you can add volatile=True option to the attribute to prevent this errors
Matthew
Is it possible that it was updated by pony, in another session (another process) ?
Alexander
Yes, this is the main possible reason specified in the message. But sometimes you can get this error when update happens in a current transaction too
Matthew
What exactly do you mean by current transaction? The transaction which the error is displayed from?
Matthew
Aha, it is blowing up on facebook_ad_account = models.FacebookAdAccount.get(uuid=facebook_ad_account_uuid)
Matthew
so the Account object was previously read and processed in the session
Matthew
then another session read and processed it
Matthew
then the current session loaded it again