Is there a way I can make pony show the connection string it is using to connect to postgres?

Pony does not uses connection string internally, it passes to psycopg2 argsuments that we provided in Database constructor or db.bind. You can access them in the following undocumented way: print(db.provider.pool.args, db.provider.pool.kwargs)

For getting json representation from my models I have used also the Marshmallow library. Maybe worth looking into

Thanks I’ll check it out :)

PonyJS JavaScript file is in the repository, but not included in PyPI distributive, becuase of it unreleased state. Python PonyORM modules has to_json, but it is undocumented

Some time later we will remove to_json and replace it with some new API

One last question, does the permissions_for function fall under the PonyJS code ?

yes

Ok, thanks :)

Hi, there is any way to have the database models on a separated file? There is an example?

https://stackoverflow.com/questions/51601838/ponyorm-multiple-model-files/51671065#51671065

Oh, thank you

Sure

Hello, sorry to bother you! I’m interested in helping out with pony, but it seems like there isn’t a CONTRIBUTING.md or many merge requests by other people. I’m guessing you’re limiting contributions to employees or something? I just want to make sure before I accidentally do something stupid.

Hi Shane! Glad you want to help! We are not limiting contributions to employees, but if a contribution is non-trivial we ask a user to sign a contributor agreement, just to be sure that the contributed code can be truly licensed under MIT/Apache 2, and doesn't contain some legal/patent-related surprises. It is a common practice, Django does the same. Saying that, it may be not an easy task to have Pony pool request accepted. Pony internals are pretty complex, and many pool requests does not implement all necessary code in a correct way. For example, adding a new data type requires not only writing appropriate converter to/from database values, but also adding corresponding monads to translate new types of expressions inside generators. Because of that, we can accept a pull request only after it was reviewed carefully by people who understands all Pony internal details. I very often was in situation when some feature looks simple, but during implementing it it gradually become clear that big additional work is necessary in caching mechanism, nested subqueries or somewhere else. So, the first impression can be deceptive. If you want to write some contribution I recommend you to initially discuss your ideas here so we can roughly estimate complexity of the task and give some advices. After the pull request is done, sometimes it worth to remind us about it, because we can be distracted with other tasks

I could help out on some of the tasks that are still outstanding from https://github.com/ponyorm/pony/issues/184#issuecomment-257295870, although that post is around 2 years old so maybe you've gotten those taken care of. I use this library a lot at work and understanding how the source code works is important. My main concern is that the source is difficult to understand and debug at certain points (eg core.py) so I would like to help with fixing that or documenting the innards more thoroughly. I hope that you don't take this the wrong way! I understand that this isn't a trivial task so I'll probably just be helping out with the backlog if I do end up helping out.

This would be great, it is always worrying to dive into the inner parts of ponyOrm

Sorry, my phone is at 1%

Splitting core.py and maybe autoformat (black?) seems a good start to me. Commenting is not an easy thing but would be very helpfull

Maybe I could start a small proof of concept branch for just core.py and start from there? I don’t think much of the stated blocking features would touch it so there won’t be too bad of a merge conflict

And if it pans out I keep it as a parallel branch until it’s fully baked, if it isn’t then I just delete the branch

In principle I'm for refactoring to PEP8. The only deviation I want to keep is to always name a variable according to its type, that is, use attr instead of self, etc. That make full-project searching of attr.something much easier, especially if there is entity.something, table.something, etc. But all other things may be changed according to PEP8. The reason why we didn't fix it yet is we have some long-standing private branches which were not merged into master yet. And if we reformat code in master, then we would not be able to do merge anymore, because of huuuge and totally unreadable merge conflicts after reformatting. I hope we can merge these branches this year, and then will be able to reformat the code

Is it possible to contribute to those branches or is it just not something you want to share before it’s ready? If it’s the latter then I understand and I’ll wait until they’re ready to be released.

I don't fully reviewed the code in these branches yet. Some things requires changes, but at this moment it is hard to say what exactly. Some time ago I came to the conclusion that it is very dangerous to accept code changes in Pony that I do not fully understand, because it can slow down further refactoring tremendously. I suggest that we return to this topic later. I hope I will have time to review these branches during this year

That makes sense, thanks. I guess I’ll check back in later.

I believe we have found a minor race condition source in Pony. Our codebase (Tribler-IPv8-Trustchain project) uses nosetests for testing, twisted for asynchronous programming and libtorrent for torrent downloads. When we run a nosetests-managed test that downloads something via Libtorrent, we get an exception in thread Twisted on a Python interpreter exit. I believe this happens because Pony's db.disconnect() does not have enough time to stop the connection before the interpreter is torn down . If sleep(1) is added after db.disconnect() the exception is much less likely to happen. It is interesting to note that running the same test through trial never crashes the interpreter. Before moving to Pony we were using our own SQLite manager based on APSW. It never showed any signs of race conditions, probably because APSW was designed to be thread-safe: https://www.reddit.com/r/Python/comments/yx20w/the_sqlite_lock_timeout_nightmare/ We would like to move our whole codebase to Pony. And that means hundreds of connections per second on a very large and diverse user base (Trustchain was chosen by the Government of Netherlands to become the base platform for the new Dutch passport system). Rock-stable performance is crucial for us. @metaprogrammer , could you please comment on Pony stability under heavy concurrent load, with asynchronous calls etc? Is it possible to move to APSW as a SQLite provider of choice for thread safety? Thanks.

Hi Grigory! So, the project in which an exception happens after db.disconnect() uses Pony with SQLite?

I believe this exception is probably not related to Pony or SQLite. Can you provide the exception text?

Pony designed to be thread-safe when each db_session is used from a single thread. sqlite3 module used for connection to SQLite database should be thread-safe too. The famous sqlite3 lock timeout problem is caused by two things, first is that SQLite uses a database-level write lock instead of more granular row-level locks, and the second is that by default standard sqlite3 connection set this lock too early. Pony mitigates the second problem by controlling the lock and setting it as later as possible. It is thread-safe, but lower transaction isolation level from "serializable" to "read committed" (just as in PostgreSQL)

I think moving to APSW will not improve concurrency, because the first problem (database-level lock) remains in APSW, and the second problem (too aggressive locking n sqlite3) was already solved by Pony.

Anyway, it is possible to add APSW support in Pony. But it is a medium-sized task which may require, say, two weeks, and at this moment we don't have time and budget for it. Another tasks looks more important. If someone are ready to sponsor adding APSW support, we can change priorities.

I have started my vacation today, and I'm going to Greece, where I will be on a sailing yacht with a minimal Internet. I will be able to return to active development starting from the September 3

Returning to you question, db.disconnect() with sqlite3 should basically clear internal SQLite buffers and close file descriptor. The should be no anything asynchronous in it.

@metaprogrammer , thank you for an extensive answer! We'll look into the problem further on our own. Goodbye and have a wonderful vacation!

Nice holidays !!

Thanks!

Hi!

First all sorry for my english! I have a question: supposed I have a table called ‘person’ with a INT field called ’age’

So in my python script i create the class:

class Person(db.Entity): age = Required(str)

(so I declare voluntarily age with string type)

So i execute the command “db.generate_mapping(check_tables=True)”

Why I have not errors with this code?

I have errors only if I make a commit

(supposed i create an istance of Person object: p = Person(age=“t24”)…the object is created but commit fails)

I would like receive an error when I declare “age = Required(str)” because the real type of ‘age’ field is INT. According to check_tables role: “check_tables (bool) – when True, Pony makes a simple check that the table names and attribute names in the database correspond to entities declaration. It doesn’t catch situations when the table has extra columns or when the type of particular column doesn’t match. ”

Thanks!

Hmm.. so I’m not entirely certain but I think that check would only happen if you have “field = Required(int)”. That is pony would not check the type in the underlying database if the table already exists. There is a flag to indicate if a table already exist, perhaps setting that might change things for you. Also if you’re using SQLite it stores everything as whatever data type you give it so the check wouldn’t matter. On other stricter databases this would matter and might throw an error.

I use Mysql, for this reason it’s important for me this question 🙂

I use Mysql, for this reason it’s important for me this question 🙂

I fell asleep on you. I’m just another user so won’t be much help beyond what I said already.

Is she Alexander's replacement during his hollidays? 😂😂

She looks like an expert in PonyORM

@metaprogrammer @akozlovsky is it possible to make this group private, kick the bots then have a new group for genuine pony users to be filtered through?

I'll try to do this after returning from my trip

Hi everyone! How do you deal with error like following: pony.orm.core.UnrepeatableReadError: Value of VirtualServer.discount for **** was updated outside of current transaction (was: Decimal('0.84999999999999997779553950749686919152736663818359375'), now: Decimal('0.8500')) I though this trouble can appear only if using a float field, not Decimal one. But recently (after upgrading to mysql 8) I encountered it again. Field definition is: discount = Required(Decimal, 10, 4, default=1, min=0, max=1), so it should be quantized to 4 digits. Is there only way to set volatile=True or optimistic=False?

https://docs.ponyorm.com/api_reference.html#volatile-option

Yes, I know it. I just noticed following in docs: optimistic (bool) True means this attribute will be used for automatic optimistic checks, see Optimistic concurrency control section. By default, this option is set to True for all attributes except attributes of float type - for float type attributes it is set to False by default. @akozlovsky maybe you should set optimistic to False for Decimals too?

Hi Vitaliy! Are you sure the column in the database is declared as DECIMAL and not as REAL?

Hi Alexander! Just checked: price decimal(10,2) NOT NULL, discount decimal(10,4) NOT NULL

How are the migration unittest run? This works really well in the main orm branch: PYTHONPATH=../ponytest/ python3 -m unittest discover -v but in orm-migrations pony/orm/examples/test_operations.py causes a failure (see https://github.com/ponyorm/pony/pull/377) even though I don't think it is meant to be a test and pony/tests/test_migrations/ doesn't seem to be found.

Is this a ico group?

no

Hey! Just a question, can i put new @property inside a created db.Entity, or i can put inheriting a class (like a method)?

Awesome library!

Hola! Yes you can.

Hola! Yes you can.

Only put a new property? okey, thanks!

If I get you correctly you can do like this: from pony.orm import * db = Database('sqlite', ':memory:') class Test(db.Entity): a = Required(str) db.generate_mapping(create_tables=True) def foo(self): return 52 Test.foo = property(foo) with db_session: t = Test(a='123') commit() print(t.foo) select(t for t in Test if t.foo == 52).show()

And what do you mean by put inheriting a class (like a method)?

If I get you correctly you can do like this: from pony.orm import * db = Database('sqlite', ':memory:') class Test(db.Entity): a = Required(str) db.generate_mapping(create_tables=True) def foo(self): return 52 Test.foo = property(foo) with db_session: t = Test(a='123') commit() print(t.foo) select(t for t in Test if t.foo == 52).show()

i didn't know the property method, cool.

And what do you mean by put inheriting a class (like a method)?

class ProductMixin(object): def get_name_and_price(self): return "%s (%s)" % (self.name, self.price) class Product(db.Entity, ProductMixin): name = Required(str, unique=True) price = Required(Decimal)

like the example of the official doc

but with @property

but.. thinking it better, it has no sense

xD

Okay. It will work both ways.

cool, thanks a lot!

omg, it works with aaaaall options, awesome xD

You are welcome.

Hello. I have a question about functions in queries. I understand that I can't use my own functions in them – that wouldn't make sense anyway. There is list of functions that can be used in queries in the API documentation, but there is one thing that bothers me. On the "Queries" page of the documentation there is this example: `select(o for o in Order if o.customer in select(c for c in Customer if c.name.startswith('A')))[:]`. So there is a startswith function of str. So, which functions can I really use? Is there a complete list somewhere?

First of all: https://docs.ponyorm.com/queries.html#functions-which-can-be-used-inside-a-query